[Met_help] [rt.rap.ucar.edu #89921] History for Re: PB2NC time summary questions
Howard Soh via RT
met_help at ucar.edu
Mon Jul 29 08:18:56 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi, everyone,
I am attaching this to this long thread regarding the daily max of ozone
verification, so we remember what we were doing back then.
I am now running this again using MET 8.0, and something seems to not be
working when I run point_stat. I get this error when running a daily max
of ozone:
ERROR :
ERROR : StringArray::operator[](int) const -> range check error!
ERROR :
I am using my own scripts to run this using MET, so this is not using
METplus.
You can find a log file of this here:
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430. It
runs pb2nc and point_stat for max ozone. The pb2nc code is working fine, I
believe.
The script I am using to run this is here:
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
The prepbufr files are:
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
The model file is here:
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
What could be causing this error?
Thanks!
Perry
On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <johnhg at ucar.edu> wrote:
> Great, glad that did the trick.
>
> John
>
> On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> perry.shafran at noaa.gov> wrote:
>
>> Well this worked! And Jeff confirmed to me that the 8-hr average
>> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr avg.
>>
>> Thanks!
>>
>> Perry
>>
>> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway <johnhg at ucar.edu>
>> wrote:
>>
>>> Perry,
>>>
>>> I think the problem is in your setting of valid_time="${CUR_VX_TIME}";
>>>
>>> Here's your wgrib2 output:
>>> wgrib2 oz_DAY1
>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
>>>
>>> The "d=2018081112" tells you the model initialization time. The valid
>>> time = init time + forecast time. So the OZMAX1 data is valid 16 hours
>>> later while the OZMAX8 data is valid 22 hours later. Rerun that wgrib2
>>> command using the "-vt" argument to list the verification time.
>>>
>>> Point-Stat isn't finding any OZMAX8 data because that data isn't valid
>>> at the time you're requesting.
>>>
>>> John
>>>
>>>
>>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate <
>>> perry.shafran at noaa.gov> wrote:
>>>
>>>> Hi, John,
>>>>
>>>> I haven't been able to work on this for awhile, but now I'm working on
>>>> it again but seem to be running into issues.
>>>>
>>>> I have a file that has both the daily max for 1-hr ozone and the daily
>>>> max for 8-hr ozone. The wgrib2 output looks like this:
>>>>
>>>> 9 (t10a2) /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
>>>> wgrib2 oz_DAY1
>>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
>>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
>>>> 10 (t10a2) /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
>>>>
>>>> As you can see they have different forecast times. But when I run
>>>> point_stat, I get this output:
>>>>
>>>> DEBUG 2:
>>>> DEBUG 2:
>>>> --------------------------------------------------------------------------------
>>>> DEBUG 2:
>>>> DEBUG 2: Reading data for OZMAX8/L1.
>>>> WARNING:
>>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching records
>>>> found for 'OZMAX8/L1'
>>>> WARNING:
>>>> WARNING:
>>>> WARNING: process_fcst_climo_files() -> no fields matching OZMAX8/L1
>>>> found in file: oz_DAY1
>>>> WARNING:
>>>>
>>>> Here is the relevant portion of the point_stat config file:
>>>>
>>>> fcst = {
>>>> field = [
>>>> { name = "OZMAX1"; level = "L1"; valid_time="${CUR_VX_TIME}"; },
>>>> { name = "OZMAX8"; level = "L1"; valid_time="${CUR_VX_TIME}"; }
>>>> ];
>>>> }
>>>>
>>>> obs = {
>>>> convert(x) = x * 10^9;
>>>> field = [
>>>> { name = "COPO"; level = "A1"; message_type =
>>>> "AIRNOW_MAX_230000"; },
>>>> { name = "COPO"; level = "A8"; message_type =
>>>> "AIRNOW_MAX_230000"; }
>>>> ];
>>>> }
>>>>
>>>> I don't remember if I had this working previously for both the 1hr and
>>>> the 8hr ozone, but apparently there is some sort of issue for the 8hr
>>>> ozone. Have any thoughts?
>>>>
>>>> Thanks!
>>>>
>>>> Perry
>>>>
>>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA Affiliate <
>>>> perry.shafran at noaa.gov> wrote:
>>>>
>>>>> I chose option 2, and that seems to work - I have a .stat file with
>>>>> output. Now I'll need to do some work here to see if the output looks OK,
>>>>> and to configure the verification so it verifies OZMAX1 and OZMAX8.
>>>>>
>>>>> I also wonder if this would be easily modified for PM.
>>>>>
>>>>> Perry
>>>>>
>>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway <johnhg at ucar.edu>
>>>>> wrote:
>>>>>
>>>>>> Perry,
>>>>>>
>>>>>> Yep, you've configured/run PB2NC to compute time summaries that are
>>>>>> valid at 04Z. And you're comparing them to forecasts that are purportedly
>>>>>> valid at 05Z. And that's the reason why you have so many obs rejected due
>>>>>> to the valid time.
>>>>>>
>>>>>> It's up to you guys to decide what the "correct" way of handling this
>>>>>> is... but here are a couple of options:
>>>>>>
>>>>>> (1) You could use the existing setup and address this logic in
>>>>>> Point-Stat config file. Change the matching time obs_window from 0 to:
>>>>>>
>>>>>> obs_window = {
>>>>>> beg = 3600;
>>>>>> end = 3600;
>>>>>> }
>>>>>> This tells Point-Stat to match observations which are exactly 1-hour
>>>>>> prior to the valid time of the forecast.
>>>>>>
>>>>>> (2) You could rerun PB2NC to compute a time summary from 05Z to 05Z
>>>>>> instead and then leave the Point-Stat obs_window setting at 0:
>>>>>>
>>>>>> time_summary = {
>>>>>> flag = TRUE;
>>>>>> beg = "050000";
>>>>>> end = "050000";
>>>>>> ...
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA Affiliate <
>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>
>>>>>>> Oops, spoke too soon. There *is* a .stat output file, but it's
>>>>>>> empty. Here's the output from point_stat:
>>>>>>>
>>>>>>> DEBUG 4:
>>>>>>> DEBUG 4: Data plane information:
>>>>>>> DEBUG 4: plane min: 13.86
>>>>>>> DEBUG 4: plane max: 130.82
>>>>>>> DEBUG 4: scan mode: 64
>>>>>>> DEBUG 4: valid time: 20180709_050000
>>>>>>> DEBUG 4: lead time: 170000
>>>>>>> DEBUG 4: init time: 20180708_120000
>>>>>>> DEBUG 4: bitmap flag: 255
>>>>>>>
>>>>>>>
>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for observation type
>>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation method NEAREST(1),
>>>>>>> using 0 pairs.
>>>>>>> DEBUG 3: Number of matched pairs = 0
>>>>>>> DEBUG 3: Observations processed = 124637
>>>>>>> DEBUG 3: Rejected: SID exclusion = 0
>>>>>>> DEBUG 3: Rejected: obs type = 0
>>>>>>> DEBUG 3: Rejected: valid time = 122154
>>>>>>> DEBUG 3: Rejected: bad obs value = 0
>>>>>>> DEBUG 3: Rejected: off the grid = 70
>>>>>>> DEBUG 3: Rejected: level mismatch = 1224
>>>>>>> DEBUG 3: Rejected: quality marker = 0
>>>>>>> DEBUG 3: Rejected: message type = 1189
>>>>>>> DEBUG 3: Rejected: masking region = 0
>>>>>>> DEBUG 3: Rejected: bad fcst value = 0
>>>>>>> DEBUG 3: Rejected: duplicates = 0
>>>>>>>
>>>>>>> Looks like most rejections were due to valid time.
>>>>>>>
>>>>>>> Have any thoughts about this one?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Perry
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA Affiliate <
>>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>>
>>>>>>>> Hi, John,
>>>>>>>>
>>>>>>>> Looks like the changing of the time stamp from _04 to _05 did the
>>>>>>>> trick as point_stat ran without an issue with that change.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Perry
>>>>>>>>
>>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>> Perry,
>>>>>>>>>
>>>>>>>>> I agree, there's likely some issue with requesting the timestamp
>>>>>>>>> of the data from that file. I think switching the verification time to
>>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
>>>>>>>>>
>>>>>>>>> Looks like wgrib2 and MET both agree on the initialization times
>>>>>>>>> and valid times for this data. Here's some useful wgrib2 and MET commands
>>>>>>>>> to illustrate.
>>>>>>>>>
>>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast time
>>>>>>>>> (-ftime), and valid time (-vt). Here it is using a sample data file you
>>>>>>>>> sent to me. But remember I'm still using the "old" wgrib2 so I have that
>>>>>>>>> funky forecast hour:
>>>>>>>>>
>>>>>>>>> wgrib2 -t -ftime -vt aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
>>>>>>>>> fcst:vt=2018070505
>>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
>>>>>>>>>
>>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity level (-v
>>>>>>>>> 4) to have MET tell you the timing info of the data it is reading:
>>>>>>>>>
>>>>>>>>> # Read the first record
>>>>>>>>> plot_data_plane aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
>>>>>>>>> ...
>>>>>>>>> DEBUG 4: valid time: 20180705_050000
>>>>>>>>> DEBUG 4: lead time: 230000
>>>>>>>>> DEBUG 4: init time: 20180704_060000
>>>>>>>>> ...
>>>>>>>>> # Read the second record
>>>>>>>>> plot_data_plane aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
>>>>>>>>> ...
>>>>>>>>> DEBUG 4: valid time: 20180706_050000
>>>>>>>>> DEBUG 4: lead time: 470000
>>>>>>>>> DEBUG 4: init time: 20180704_060000
>>>>>>>>>
>>>>>>>>> Hope that helps.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA Affiliate <
>>>>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, John,
>>>>>>>>>>
>>>>>>>>>> When I implemented all the changes (and also set it up so it's
>>>>>>>>>> reading the beta6 version), I used this command:
>>>>>>>>>>
>>>>>>>>>> CUR_VX_TIME=20180709_04
>>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
>>>>>>>>>> /meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max -v 4
>>>>>>>>>>
>>>>>>>>>> But then I got this error:
>>>>>>>>>>
>>>>>>>>>> DEBUG 2:
>>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
>>>>>>>>>> WARNING:
>>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
>>>>>>>>>> records found for 'OZMAX1/L1'
>>>>>>>>>> WARNING:
>>>>>>>>>> WARNING:
>>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields matching
>>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
>>>>>>>>>> WARNING:
>>>>>>>>>> ERROR :
>>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested forecast data
>>>>>>>>>> found! Exiting...
>>>>>>>>>>
>>>>>>>>>> Not sure why that is. A look at the wgrib2 output for the file
>>>>>>>>>> in this case reads:
>>>>>>>>>>
>>>>>>>>>> 14 (t14a2)
>>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 > wgrib2 oz1hr_DAY1
>>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave fcst:
>>>>>>>>>> 15 (t14a2) /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>>>>>>>>>>
>>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had suggested
>>>>>>>>>> that it should be the time stamp of the 2nd day, but maybe it needs to be
>>>>>>>>>> the time stamp of the first day?
>>>>>>>>>>
>>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
>>>>>>>>>>
>>>>>>>>>> Thanks for any thoughts.
>>>>>>>>>>
>>>>>>>>>> Perry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> Perry,
>>>>>>>>>>>
>>>>>>>>>>> I'm confused. In the previous air quality vx we did, we used
>>>>>>>>>>> observation files named:
>>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
>>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
>>>>>>>>>>>
>>>>>>>>>>> And those files don't exist in the July data you just sent.
>>>>>>>>>>> Instead, it's BUFR files with names like:
>>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d aqm.t12z.airnow.tm024.bufr_d
>>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
>>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure if you're actually verifying against the BUFR or
>>>>>>>>>>> PREPBUFR.
>>>>>>>>>>>
>>>>>>>>>>> Let me just tell you the key points for the config file
>>>>>>>>>>> settings...
>>>>>>>>>>>
>>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input files -
>>>>>>>>>>> thecurrent day and the previous day...
>>>>>>>>>>> pb2nc \
>>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
>>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
>>>>>>>>>>> PB2NCConfig \
>>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>
>>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary like this:
>>>>>>>>>>> time_summary = {
>>>>>>>>>>> flag = TRUE;
>>>>>>>>>>> beg = "040000";
>>>>>>>>>>> end = "040000";
>>>>>>>>>>> step = 24*3600;
>>>>>>>>>>> width = { beg = -23*3600; end = 0; } // previous
>>>>>>>>>>> 23-hours (05Z to 04Z)
>>>>>>>>>>> grib_code = [];
>>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
>>>>>>>>>>> type = [ "max", "mean" ];
>>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
>>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when at least
>>>>>>>>>>> 75% of expect obs are present
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> That'll compute the MAX and MEAN values at each station over the
>>>>>>>>>>> defined time period. They're identified in the output by the message type:
>>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
>>>>>>>>>>>
>>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and "obs"
>>>>>>>>>>> sections like this for the max of the 1-hour data. Instead of specifying
>>>>>>>>>>> "lead_time" to filter the data, it may be more convenient to specify the
>>>>>>>>>>> valid time. So that you don't have to modify the config file, I'd suggest
>>>>>>>>>>> using environment variables:
>>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
>>>>>>>>>>>
>>>>>>>>>>> fcst = {
>>>>>>>>>>> field = [
>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
>>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
>>>>>>>>>>> ];
>>>>>>>>>>> }
>>>>>>>>>>> obs = {
>>>>>>>>>>> convert(x) = x * 10^9;
>>>>>>>>>>> field = [
>>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
>>>>>>>>>>> "AIRNOW_MAX_230000"; }
>>>>>>>>>>> ];
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sounds good, thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA Affiliate <
>>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids and focus on
>>>>>>>>>>>>> the non-bc files. It takes some time to copy files over to Theia so we'll
>>>>>>>>>>>>> let him work on the files he has for now and if he runs into problems, I'll
>>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr max/avg files).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not for bias
>>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe, you may want to provide
>>>>>>>>>>>>>> him para5 runs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for John to do
>>>>>>>>>>>>>> some testing some production files. Do you think I should use parallels
>>>>>>>>>>>>>> instead or do you think the ops files are OK for creating a MET+ use case
>>>>>>>>>>>>>> for AQ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know your thoughts. I think that the ops files should
>>>>>>>>>>>>>> be OK for now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels. We
>>>>>>>>>>>>>>> wanted to update the error in daily max hour labels but management did not
>>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
>>>>>>>>>>>>>>> Jeff
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA Affiliate wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do any of the current parallels have the correct time
>>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John over parallel CMAQ data
>>>>>>>>>>>>>>> rather than production CMAQ.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't think that the MET team has any other versions of
>>>>>>>>>>>>>>> wgrib2 than the one they already have - but they can also clarify this as
>>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does over the current wgrib2?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
>>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to use the
>>>>>>>>>>>>>>>> latest parallel run output (para5).
>>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall Jianping also
>>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management did not want to risk any
>>>>>>>>>>>>>>>> further delays so we will have to do so with the FYQ4 implementation.
>>>>>>>>>>>>>>>> Jeff
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA Affiliate wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I think I
>>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time ago, but I don't know if I
>>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I think the parallel CMAQ they
>>>>>>>>>>>>>>>> are running corrects that, but not too entirely sure. But he would have
>>>>>>>>>>>>>>>> more insight as to what is correct and how to approach this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John talks
>>>>>>>>>>>>>>>> about?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley Gotway <
>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Perry,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to compute the
>>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-hour ozone concentrations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I ran pb2nc like this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
>>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00 pb2nc_output.nc
>>>>>>>>>>>>>>>>> PB2NCConfig \
>>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And then I ran point_stat like this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat \
>>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
>>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
>>>>>>>>>>>>>>>>> -outdir point_stat -v 4
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Using the attached config files.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
>>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
>>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region FULL, for interpolation
>>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But I noticed some oddities:
>>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of this "max"
>>>>>>>>>>>>>>>>> file:
>>>>>>>>>>>>>>>>> wgrib2 aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
>>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
>>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
>>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-40 hour
>>>>>>>>>>>>>>>>> ave fcst:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing revealed
>>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours 17 and 41, respectively.
>>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the same file, we
>>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which one to use. Do this by
>>>>>>>>>>>>>>>>> defining the "lead_time":
>>>>>>>>>>>>>>>>> fcst = {
>>>>>>>>>>>>>>>>> field = [
>>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1"; lead_time =
>>>>>>>>>>>>>>>>> "41";' }
>>>>>>>>>>>>>>>>> ];
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are off by 1
>>>>>>>>>>>>>>>>> hour.
>>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead time =
>>>>>>>>>>>>>>>>> valid time of 2018052805.
>>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max value from
>>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z while the obs are valid at 04Z.
>>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat config file by
>>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
>>>>>>>>>>>>>>>>> obs_window = {
>>>>>>>>>>>>>>>>> beg = -3600;
>>>>>>>>>>>>>>>>> end = -3600;
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So what're the next steps?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh <hsoh at ucar.edu>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We're making progress on the time summaries, but I found
>>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota here:
>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute 24-hour
>>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the output to an ascii file using
>>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to count up the number of regular
>>>>>>>>>>>>>>>>>> and maximum obs values:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt | wc -l
>>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt | sort -u
>>>>>>>>>>>>>>>>>> | wc -l
>>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt | wc -l
>>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt | sort -u |
>>>>>>>>>>>>>>>>>> wc -l
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that all of
>>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555 meaning
>>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are duplicates.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For example, the last station ID in that file is named
>>>>>>>>>>>>>>>>>> "2FB31EEC":
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep 2FB31EEC
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in the last two
>>>>>>>>>>>>>>>>>> lines.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Could you please check to see why these duplicates are
>>>>>>>>>>>>>>>>>> showing up?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh <hsoh at ucar.edu>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> No, I don't have them yet.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers.
>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why we'd have
>>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across 2 files. Do we have existing
>>>>>>>>>>>>>>>>>>> examples of smaller files from which we could compute a 24-hour max
>>>>>>>>>>>>>>>>>>> temperature or something?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes (550.319 sec).
>>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
>>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in the new
>>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh <hsoh at ucar.edu>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content of obs_hid
>>>>>>>>>>>>>>>>>>>>> variable).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary (test
>>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
>>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I added one
>>>>>>>>>>>>>>>>>>>>> with time summary and two input files and will check the unit test today.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change the nightly
>>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently covered? If not, should I add
>>>>>>>>>>>>>>>>>>>>> it as a new unit test?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary data.
>>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The output is available
>>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
>>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley Gotway <
>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic for PB2NC.
>>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-hour maximum values of COPO from
>>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr counts:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to dump to ascii
>>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000 string show up in the output.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC not actually
>>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the Rscript dumping the data.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
>>>>>>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John Halley Gotway <
>>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard Soh <
>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
>>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and 57246 observations
>>>>>>>>>>>>>>>>>>>>>>>>> from aqm.20180528/aqm.t12z.prepbufr.tm00. There
>>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data. I'm working on this.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John Halley Gotway
>>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please take a
>>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC. I copied the data over to
>>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
>>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
>>>>>>>>>>>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
>>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly maximums
>>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
>>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
>>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
>>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
>>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
>>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end = 0; } //
>>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
>>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
>>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
>>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
>>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
>>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but using
>>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what you'd expect. The hdr_typ_table
>>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the "AIRNOW_MAX_240000" message type, but as you pointed
>>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused by using 2
>>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them to cover the desired 04Z to 04Z
>>>>>>>>>>>>>>>>>>>>>>>>>> time window.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John Halley
>>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW PrepBUFR file
>>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24 hour maximum of obs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
>>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is reasonable for me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data in NetCDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of "AIRNOW_MAX_240000".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was generated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without any
>>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers than obs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF output:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295 currently)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to 3655.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for summary) exists
>>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable. The first header index for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than 3655.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
>>>>>>>>>>>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ndas.20120409.t12z.prepbufr.tm00.summary.nc
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1 -d" " | sort
>>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota in here:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180622
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to ascii:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
>>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of them should actually be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by this:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table pb2nc_output.nc
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table lookup
>>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Fri Apr 26 10:08:38 2019
Hi Perry -
I'm also forwarding this ticket to Howard,
I think John is quite busy with the MET 8.1 release so it might be
later
next week before he gets a chance to look at this.
If you want to upload your data and config files to our ftp server we
can
replicate and debug the error here.
I can have a first look this afternoon.
You can create a subdir at
ftp.rap.ucar.edu://incoming/irap/met_help
thanks,
David
On Thu, Apr 25, 2019 at 11:17 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Thu Apr 25 11:16:42 2019: Request 89921 was acted upon.
> Transaction: Ticket created by perry.shafran at noaa.gov
> Queue: met_help
> Subject: Re: PB2NC time summary questions
> Owner: Nobody
> Requestors: perry.shafran at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
>
> Hi, everyone,
>
> I am attaching this to this long thread regarding the daily max of
ozone
> verification, so we remember what we were doing back then.
>
> I am now running this again using MET 8.0, and something seems to
not be
> working when I run point_stat. I get this error when running a
daily max
> of ozone:
>
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> I am using my own scripts to run this using MET, so this is not
using
> METplus.
>
> You can find a log file of this here:
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
It
> runs pb2nc and point_stat for max ozone. The pb2nc code is working
fine, I
> believe.
>
> The script I am using to run this is here:
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
>
> The prepbufr files are:
>
> /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
>
> The model file is here:
>
>
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
>
>
> What could be causing this error?
>
> Thanks!
>
> Perry
>
>
> On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Great, glad that did the trick.
> >
> > John
> >
> > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > perry.shafran at noaa.gov> wrote:
> >
> >> Well this worked! And Jeff confirmed to me that the 8-hr average
> >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
avg.
> >>
> >> Thanks!
> >>
> >> Perry
> >>
> >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
<johnhg at ucar.edu>
> >> wrote:
> >>
> >>> Perry,
> >>>
> >>> I think the problem is in your setting of
valid_time="${CUR_VX_TIME}";
> >>>
> >>> Here's your wgrib2 output:
> >>> wgrib2 oz_DAY1
> >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> >>>
> >>> The "d=2018081112" tells you the model initialization time. The
valid
> >>> time = init time + forecast time. So the OZMAX1 data is valid
16 hours
> >>> later while the OZMAX8 data is valid 22 hours later. Rerun that
wgrib2
> >>> command using the "-vt" argument to list the verification time.
> >>>
> >>> Point-Stat isn't finding any OZMAX8 data because that data isn't
valid
> >>> at the time you're requesting.
> >>>
> >>> John
> >>>
> >>>
> >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate <
> >>> perry.shafran at noaa.gov> wrote:
> >>>
> >>>> Hi, John,
> >>>>
> >>>> I haven't been able to work on this for awhile, but now I'm
working on
> >>>> it again but seem to be running into issues.
> >>>>
> >>>> I have a file that has both the daily max for 1-hr ozone and
the daily
> >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> >>>>
> >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> >>>> wgrib2 oz_DAY1
> >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
> >>>> 10 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> >>>>
> >>>> As you can see they have different forecast times. But when I
run
> >>>> point_stat, I get this output:
> >>>>
> >>>> DEBUG 2:
> >>>> DEBUG 2:
> >>>>
>
--------------------------------------------------------------------------------
> >>>> DEBUG 2:
> >>>> DEBUG 2: Reading data for OZMAX8/L1.
> >>>> WARNING:
> >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
records
> >>>> found for 'OZMAX8/L1'
> >>>> WARNING:
> >>>> WARNING:
> >>>> WARNING: process_fcst_climo_files() -> no fields matching
OZMAX8/L1
> >>>> found in file: oz_DAY1
> >>>> WARNING:
> >>>>
> >>>> Here is the relevant portion of the point_stat config file:
> >>>>
> >>>> fcst = {
> >>>> field = [
> >>>> { name = "OZMAX1"; level = "L1";
valid_time="${CUR_VX_TIME}"; },
> >>>> { name = "OZMAX8"; level = "L1";
valid_time="${CUR_VX_TIME}"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> obs = {
> >>>> convert(x) = x * 10^9;
> >>>> field = [
> >>>> { name = "COPO"; level = "A1"; message_type =
> >>>> "AIRNOW_MAX_230000"; },
> >>>> { name = "COPO"; level = "A8"; message_type =
> >>>> "AIRNOW_MAX_230000"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> I don't remember if I had this working previously for both the
1hr and
> >>>> the 8hr ozone, but apparently there is some sort of issue for
the 8hr
> >>>> ozone. Have any thoughts?
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Perry
> >>>>
> >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA Affiliate
<
> >>>> perry.shafran at noaa.gov> wrote:
> >>>>
> >>>>> I chose option 2, and that seems to work - I have a .stat file
with
> >>>>> output. Now I'll need to do some work here to see if the
output
> looks OK,
> >>>>> and to configure the verification so it verifies OZMAX1 and
OZMAX8.
> >>>>>
> >>>>> I also wonder if this would be easily modified for PM.
> >>>>>
> >>>>> Perry
> >>>>>
> >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
<johnhg at ucar.edu
> >
> >>>>> wrote:
> >>>>>
> >>>>>> Perry,
> >>>>>>
> >>>>>> Yep, you've configured/run PB2NC to compute time summaries
that are
> >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> purportedly
> >>>>>> valid at 05Z. And that's the reason why you have so many obs
> rejected due
> >>>>>> to the valid time.
> >>>>>>
> >>>>>> It's up to you guys to decide what the "correct" way of
handling
> this
> >>>>>> is... but here are a couple of options:
> >>>>>>
> >>>>>> (1) You could use the existing setup and address this logic
in
> >>>>>> Point-Stat config file. Change the matching time obs_window
from 0
> to:
> >>>>>>
> >>>>>> obs_window = {
> >>>>>> beg = 3600;
> >>>>>> end = 3600;
> >>>>>> }
> >>>>>> This tells Point-Stat to match observations which are exactly
1-hour
> >>>>>> prior to the valid time of the forecast.
> >>>>>>
> >>>>>> (2) You could rerun PB2NC to compute a time summary from 05Z
to 05Z
> >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> >>>>>>
> >>>>>> time_summary = {
> >>>>>> flag = TRUE;
> >>>>>> beg = "050000";
> >>>>>> end = "050000";
> >>>>>> ...
> >>>>>>
> >>>>>> Thanks,
> >>>>>> John
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate <
> >>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>
> >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
it's
> >>>>>>> empty. Here's the output from point_stat:
> >>>>>>>
> >>>>>>> DEBUG 4:
> >>>>>>> DEBUG 4: Data plane information:
> >>>>>>> DEBUG 4: plane min: 13.86
> >>>>>>> DEBUG 4: plane max: 130.82
> >>>>>>> DEBUG 4: scan mode: 64
> >>>>>>> DEBUG 4: valid time: 20180709_050000
> >>>>>>> DEBUG 4: lead time: 170000
> >>>>>>> DEBUG 4: init time: 20180708_120000
> >>>>>>> DEBUG 4: bitmap flag: 255
> >>>>>>>
> >>>>>>>
> >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation type
> >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> NEAREST(1),
> >>>>>>> using 0 pairs.
> >>>>>>> DEBUG 3: Number of matched pairs = 0
> >>>>>>> DEBUG 3: Observations processed = 124637
> >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> >>>>>>> DEBUG 3: Rejected: obs type = 0
> >>>>>>> DEBUG 3: Rejected: valid time = 122154
> >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> >>>>>>> DEBUG 3: Rejected: off the grid = 70
> >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> >>>>>>> DEBUG 3: Rejected: quality marker = 0
> >>>>>>> DEBUG 3: Rejected: message type = 1189
> >>>>>>> DEBUG 3: Rejected: masking region = 0
> >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> >>>>>>> DEBUG 3: Rejected: duplicates = 0
> >>>>>>>
> >>>>>>> Looks like most rejections were due to valid time.
> >>>>>>>
> >>>>>>> Have any thoughts about this one?
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> Perry
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
Affiliate <
> >>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>
> >>>>>>>> Hi, John,
> >>>>>>>>
> >>>>>>>> Looks like the changing of the time stamp from _04 to _05
did the
> >>>>>>>> trick as point_stat ran without an issue with that change.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Perry
> >>>>>>>>
> >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> >>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>
> >>>>>>>>> Perry,
> >>>>>>>>>
> >>>>>>>>> I agree, there's likely some issue with requesting the
timestamp
> >>>>>>>>> of the data from that file. I think switching the
verification
> time to
> >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> >>>>>>>>>
> >>>>>>>>> Looks like wgrib2 and MET both agree on the initialization
times
> >>>>>>>>> and valid times for this data. Here's some useful wgrib2
and
> MET commands
> >>>>>>>>> to illustrate.
> >>>>>>>>>
> >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
time
> >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a sample
data
> file you
> >>>>>>>>> sent to me. But remember I'm still using the "old" wgrib2
so I
> have that
> >>>>>>>>> funky forecast hour:
> >>>>>>>>>
> >>>>>>>>> wgrib2 -t -ftime -vt
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> >>>>>>>>> fcst:vt=2018070505
> >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
> >>>>>>>>>
> >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level (-v
> >>>>>>>>> 4) to have MET tell you the timing info of the data it is
> reading:
> >>>>>>>>>
> >>>>>>>>> # Read the first record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> >>>>>>>>> DEBUG 4: lead time: 230000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>> ...
> >>>>>>>>> # Read the second record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> >>>>>>>>> DEBUG 4: lead time: 470000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>>
> >>>>>>>>> Hope that helps.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> John
> >>>>>>>>>
> >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
Affiliate <
> >>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi, John,
> >>>>>>>>>>
> >>>>>>>>>> When I implemented all the changes (and also set it up so
it's
> >>>>>>>>>> reading the beta6 version), I used this command:
> >>>>>>>>>>
> >>>>>>>>>> CUR_VX_TIME=20180709_04
> >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> >>>>>>>>>>
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
-v 4
> >>>>>>>>>>
> >>>>>>>>>> But then I got this error:
> >>>>>>>>>>
> >>>>>>>>>> DEBUG 2:
> >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> >>>>>>>>>> records found for 'OZMAX1/L1'
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields matching
> >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> >>>>>>>>>> WARNING:
> >>>>>>>>>> ERROR :
> >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> data
> >>>>>>>>>> found! Exiting...
> >>>>>>>>>>
> >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the file
> >>>>>>>>>> in this case reads:
> >>>>>>>>>>
> >>>>>>>>>> 14 (t14a2)
> >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> wgrib2 oz1hr_DAY1
> >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> >>>>>>>>>> 15 (t14a2)
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>>>>>>>>
> >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
suggested
> >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe it
> needs to be
> >>>>>>>>>> the time stamp of the first day?
> >>>>>>>>>>
> >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for any thoughts.
> >>>>>>>>>>
> >>>>>>>>>> Perry
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> >>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Perry,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm confused. In the previous air quality vx we did, we
used
> >>>>>>>>>>> observation files named:
> >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> >>>>>>>>>>>
> >>>>>>>>>>> And those files don't exist in the July data you just
sent.
> >>>>>>>>>>> Instead, it's BUFR files with names like:
> >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure if you're actually verifying against the
BUFR or
> >>>>>>>>>>> PREPBUFR.
> >>>>>>>>>>>
> >>>>>>>>>>> Let me just tell you the key points for the config file
> >>>>>>>>>>> settings...
> >>>>>>>>>>>
> >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files -
> >>>>>>>>>>> thecurrent day and the previous day...
> >>>>>>>>>>> pb2nc \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> >>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>
> >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
like
> this:
> >>>>>>>>>>> time_summary = {
> >>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>> beg = "040000";
> >>>>>>>>>>> end = "040000";
> >>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> >>>>>>>>>>> 23-hours (05Z to 04Z)
> >>>>>>>>>>> grib_code = [];
> >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> >>>>>>>>>>> type = [ "max", "mean" ];
> >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at least
> >>>>>>>>>>> 75% of expect obs are present
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>> That'll compute the MAX and MEAN values at each station
over
> the
> >>>>>>>>>>> defined time period. They're identified in the output
by the
> message type:
> >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> >>>>>>>>>>>
> >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead of
> specifying
> >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient to
> specify the
> >>>>>>>>>>> valid time. So that you don't have to modify the config
file,
> I'd suggest
> >>>>>>>>>>> using environment variables:
> >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> >>>>>>>>>>>
> >>>>>>>>>>> fcst = {
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>> obs = {
> >>>>>>>>>>> convert(x) = x * 10^9;
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> >>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Sounds good, thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> John
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> Affiliate <
> >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids and
focus
> on
> >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
over to
> Theia so we'll
> >>>>>>>>>>>>> let him work on the files he has for now and if he
runs into
> problems, I'll
> >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
max/avg
> files).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not for
bias
> >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
you may
> want to provide
> >>>>>>>>>>>>>> him para5 runs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John to do
> >>>>>>>>>>>>>> some testing some production files. Do you think I
should
> use parallels
> >>>>>>>>>>>>>> instead or do you think the ops files are OK for
creating a
> MET+ use case
> >>>>>>>>>>>>>> for AQ?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> should
> >>>>>>>>>>>>>> be OK for now.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
We
> >>>>>>>>>>>>>>> wanted to update the error in daily max hour labels
but
> management did not
> >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> >>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA Affiliate
wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John over
> parallel CMAQ data
> >>>>>>>>>>>>>>> rather than production CMAQ.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions of
> >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they can
also
> clarify this as
> >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does over
the
> current wgrib2?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to use
the
> >>>>>>>>>>>>>>>> latest parallel run output (para5).
> >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall Jianping
also
> >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
did not
> want to risk any
> >>>>>>>>>>>>>>>> further delays so we will have to do so with the
FYQ4
> implementation.
> >>>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA Affiliate
wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think I
> >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago, but
> I don't know if I
> >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I think
the
> parallel CMAQ they
> >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> But he would have
> >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> this.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
talks
> >>>>>>>>>>>>>>>> about?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley Gotway
<
> >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Perry,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute the
> >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
hour
> ozone concentrations.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> pb2nc_output.nc
> >>>>>>>>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
\
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
\
> >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Using the attached config files.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
FULL,
> for interpolation
> >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But I noticed some oddities:
> >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
this
> "max"
> >>>>>>>>>>>>>>>>> file:
> >>>>>>>>>>>>>>>>> wgrib2
> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-40
hour
> >>>>>>>>>>>>>>>>> ave fcst:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
revealed
> >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17 and
> 41, respectively.
> >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the same
file,
> we
> >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which one
to
> use. Do this by
> >>>>>>>>>>>>>>>>> defining the "lead_time":
> >>>>>>>>>>>>>>>>> fcst = {
> >>>>>>>>>>>>>>>>> field = [
> >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> >>>>>>>>>>>>>>>>> "41";' }
> >>>>>>>>>>>>>>>>> ];
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off by 1
> >>>>>>>>>>>>>>>>> hour.
> >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
time =
> >>>>>>>>>>>>>>>>> valid time of 2018052805.
> >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max value
from
> >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z while
the
> obs are valid at 04Z.
> >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat config
file
> by
> >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> >>>>>>>>>>>>>>>>> obs_window = {
> >>>>>>>>>>>>>>>>> beg = -3600;
> >>>>>>>>>>>>>>>>> end = -3600;
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So what're the next steps?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We're making progress on the time summaries, but
I found
> >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute 24-
hour
> >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output to
> an ascii file using
> >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to count
up the
> number of regular
> >>>>>>>>>>>>>>>>>> and maximum obs values:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
wc -l
> >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
sort
> -u
> >>>>>>>>>>>>>>>>>> | wc -l
> >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt | wc
-l
> >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
sort -u |
> >>>>>>>>>>>>>>>>>> wc -l
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
all of
> >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
meaning
> >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
duplicates.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> For example, the last station ID in that file is
named
> >>>>>>>>>>>>>>>>>> "2FB31EEC":
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
2FB31EEC
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in the
last
> two
> >>>>>>>>>>>>>>>>>> lines.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates are
> >>>>>>>>>>>>>>>>>> showing up?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers.
> >>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
we'd have
> >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across 2
> files. Do we have existing
> >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
compute
> a 24-hour max
> >>>>>>>>>>>>>>>>>>> temperature or something?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319 sec).
> >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in the
new
> >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content of
> obs_hid
> >>>>>>>>>>>>>>>>>>>>> variable).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
(test
> >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
added
> one
> >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> check the unit test today.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change the
nightly
> >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> covered? If not, should I add
> >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
data.
> >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The output is available
> >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
Gotway <
> >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic for
PB2NC.
> >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> maximum values of COPO from
> >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr counts:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to dump
to
> ascii
> >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000 string
show
> up in the output.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC not
> actually
> >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
Rscript
> dumping the data.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John Halley
Gotway
> <
> >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard Soh
<
> >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
57246
> observations
> >>>>>>>>>>>>>>>>>>>>>>>>> from aqm.20180528/aqm.t12z.prepbufr.tm00.
There
> >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
I'm
> working on this.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> Gotway
> >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
take a
> >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC. I
> copied the data over to
> >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
maximums
> >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end = 0;
} //
> >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what you'd
> expect. The hdr_typ_table
> >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the "AIRNOW_MAX_240000"
> message type, but as you pointed
> >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused by
using 2
> >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> cover the desired 04Z to 04Z
> >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW PrepBUFR
file
> >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
hour
> maximum of obs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is reasonable
for me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data in
NetCDF
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> "AIRNOW_MAX_240000".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
generated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
than obs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
output:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for summary)
exists
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable. The
first
> header index for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1 -d" "
| sort
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota in
here:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180622
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all of
the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
them
> should actually be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table pb2nc_output.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
lookup
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Apr 26 10:25:44 2019
Hi, David,
I put all you need in the directory /incoming/irap/met_help/pshafran
on
your ftp server.
Let me know if you require more items or need some explanation of what
I
have here.
Thanks!
Perry
On Fri, Apr 26, 2019 at 12:08 PM David Fillmore via RT
<met_help at ucar.edu>
wrote:
> Hi Perry -
>
> I'm also forwarding this ticket to Howard,
> I think John is quite busy with the MET 8.1 release so it might be
later
> next week before he gets a chance to look at this.
>
> If you want to upload your data and config files to our ftp server
we can
> replicate and debug the error here.
> I can have a first look this afternoon.
> You can create a subdir at
> ftp.rap.ucar.edu://incoming/irap/met_help
>
> thanks,
> David
>
> On Thu, Apr 25, 2019 at 11:17 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Apr 25 11:16:42 2019: Request 89921 was acted upon.
> > Transaction: Ticket created by perry.shafran at noaa.gov
> > Queue: met_help
> > Subject: Re: PB2NC time summary questions
> > Owner: Nobody
> > Requestors: perry.shafran at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> >
> > Hi, everyone,
> >
> > I am attaching this to this long thread regarding the daily max of
ozone
> > verification, so we remember what we were doing back then.
> >
> > I am now running this again using MET 8.0, and something seems to
not be
> > working when I run point_stat. I get this error when running a
daily max
> > of ozone:
> >
> > ERROR :
> > ERROR : StringArray::operator[](int) const -> range check error!
> > ERROR :
> >
> > I am using my own scripts to run this using MET, so this is not
using
> > METplus.
> >
> > You can find a log file of this here:
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> It
> > runs pb2nc and point_stat for max ozone. The pb2nc code is working
fine,
> I
> > believe.
> >
> > The script I am using to run this is here:
> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> >
> > The prepbufr files are:
> >
> > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> >
> > The model file is here:
> >
> >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> >
> >
> > What could be causing this error?
> >
> > Thanks!
> >
> > Perry
> >
> >
> > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Great, glad that did the trick.
> > >
> > > John
> > >
> > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > > perry.shafran at noaa.gov> wrote:
> > >
> > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
avg.
> > >>
> > >> Thanks!
> > >>
> > >> Perry
> > >>
> > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
<johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >>> Perry,
> > >>>
> > >>> I think the problem is in your setting of
> valid_time="${CUR_VX_TIME}";
> > >>>
> > >>> Here's your wgrib2 output:
> > >>> wgrib2 oz_DAY1
> > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > >>>
> > >>> The "d=2018081112" tells you the model initialization time.
The
> valid
> > >>> time = init time + forecast time. So the OZMAX1 data is valid
16
> hours
> > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> wgrib2
> > >>> command using the "-vt" argument to list the verification
time.
> > >>>
> > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> valid
> > >>> at the time you're requesting.
> > >>>
> > >>> John
> > >>>
> > >>>
> > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate
<
> > >>> perry.shafran at noaa.gov> wrote:
> > >>>
> > >>>> Hi, John,
> > >>>>
> > >>>> I haven't been able to work on this for awhile, but now I'm
working
> on
> > >>>> it again but seem to be running into issues.
> > >>>>
> > >>>> I have a file that has both the daily max for 1-hr ozone and
the
> daily
> > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > >>>>
> > >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > >>>> wgrib2 oz_DAY1
> > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > >>>> 10 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > >>>>
> > >>>> As you can see they have different forecast times. But when
I run
> > >>>> point_stat, I get this output:
> > >>>>
> > >>>> DEBUG 2:
> > >>>> DEBUG 2:
> > >>>>
> >
>
--------------------------------------------------------------------------------
> > >>>> DEBUG 2:
> > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > >>>> WARNING:
> > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
records
> > >>>> found for 'OZMAX8/L1'
> > >>>> WARNING:
> > >>>> WARNING:
> > >>>> WARNING: process_fcst_climo_files() -> no fields matching
OZMAX8/L1
> > >>>> found in file: oz_DAY1
> > >>>> WARNING:
> > >>>>
> > >>>> Here is the relevant portion of the point_stat config file:
> > >>>>
> > >>>> fcst = {
> > >>>> field = [
> > >>>> { name = "OZMAX1"; level = "L1";
valid_time="${CUR_VX_TIME}";
> },
> > >>>> { name = "OZMAX8"; level = "L1";
valid_time="${CUR_VX_TIME}";
> }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> obs = {
> > >>>> convert(x) = x * 10^9;
> > >>>> field = [
> > >>>> { name = "COPO"; level = "A1"; message_type =
> > >>>> "AIRNOW_MAX_230000"; },
> > >>>> { name = "COPO"; level = "A8"; message_type =
> > >>>> "AIRNOW_MAX_230000"; }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> I don't remember if I had this working previously for both
the 1hr
> and
> > >>>> the 8hr ozone, but apparently there is some sort of issue for
the
> 8hr
> > >>>> ozone. Have any thoughts?
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>> Perry
> > >>>>
> > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate <
> > >>>> perry.shafran at noaa.gov> wrote:
> > >>>>
> > >>>>> I chose option 2, and that seems to work - I have a .stat
file with
> > >>>>> output. Now I'll need to do some work here to see if the
output
> > looks OK,
> > >>>>> and to configure the verification so it verifies OZMAX1 and
OZMAX8.
> > >>>>>
> > >>>>> I also wonder if this would be easily modified for PM.
> > >>>>>
> > >>>>> Perry
> > >>>>>
> > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway <
> johnhg at ucar.edu
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Perry,
> > >>>>>>
> > >>>>>> Yep, you've configured/run PB2NC to compute time summaries
that
> are
> > >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> > purportedly
> > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > rejected due
> > >>>>>> to the valid time.
> > >>>>>>
> > >>>>>> It's up to you guys to decide what the "correct" way of
handling
> > this
> > >>>>>> is... but here are a couple of options:
> > >>>>>>
> > >>>>>> (1) You could use the existing setup and address this logic
in
> > >>>>>> Point-Stat config file. Change the matching time
obs_window from
> 0
> > to:
> > >>>>>>
> > >>>>>> obs_window = {
> > >>>>>> beg = 3600;
> > >>>>>> end = 3600;
> > >>>>>> }
> > >>>>>> This tells Point-Stat to match observations which are
exactly
> 1-hour
> > >>>>>> prior to the valid time of the forecast.
> > >>>>>>
> > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z to
> 05Z
> > >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> > >>>>>>
> > >>>>>> time_summary = {
> > >>>>>> flag = TRUE;
> > >>>>>> beg = "050000";
> > >>>>>> end = "050000";
> > >>>>>> ...
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> John
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate <
> > >>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>
> > >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
it's
> > >>>>>>> empty. Here's the output from point_stat:
> > >>>>>>>
> > >>>>>>> DEBUG 4:
> > >>>>>>> DEBUG 4: Data plane information:
> > >>>>>>> DEBUG 4: plane min: 13.86
> > >>>>>>> DEBUG 4: plane max: 130.82
> > >>>>>>> DEBUG 4: scan mode: 64
> > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > >>>>>>> DEBUG 4: lead time: 170000
> > >>>>>>> DEBUG 4: init time: 20180708_120000
> > >>>>>>> DEBUG 4: bitmap flag: 255
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> type
> > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > NEAREST(1),
> > >>>>>>> using 0 pairs.
> > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > >>>>>>> DEBUG 3: Observations processed = 124637
> > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > >>>>>>>
> > >>>>>>> Looks like most rejections were due to valid time.
> > >>>>>>>
> > >>>>>>> Have any thoughts about this one?
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>>
> > >>>>>>> Perry
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
Affiliate <
> > >>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>
> > >>>>>>>> Hi, John,
> > >>>>>>>>
> > >>>>>>>> Looks like the changing of the time stamp from _04 to _05
did
> the
> > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > >>>>>>>>
> > >>>>>>>> Thanks!
> > >>>>>>>>
> > >>>>>>>> Perry
> > >>>>>>>>
> > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > >>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>
> > >>>>>>>>> Perry,
> > >>>>>>>>>
> > >>>>>>>>> I agree, there's likely some issue with requesting the
> timestamp
> > >>>>>>>>> of the data from that file. I think switching the
verification
> > time to
> > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > >>>>>>>>>
> > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> times
> > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2 and
> > MET commands
> > >>>>>>>>> to illustrate.
> > >>>>>>>>>
> > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
time
> > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample data
> > file you
> > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2 so I
> > have that
> > >>>>>>>>> funky forecast hour:
> > >>>>>>>>>
> > >>>>>>>>> wgrib2 -t -ftime -vt
> > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > >>>>>>>>> fcst:vt=2018070505
> > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > >>>>>>>>>
> > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level
> (-v
> > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > reading:
> > >>>>>>>>>
> > >>>>>>>>> # Read the first record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > >>>>>>>>> DEBUG 4: lead time: 230000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>> ...
> > >>>>>>>>> # Read the second record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > >>>>>>>>> DEBUG 4: lead time: 470000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>>
> > >>>>>>>>> Hope that helps.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> John
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
Affiliate
> <
> > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, John,
> > >>>>>>>>>>
> > >>>>>>>>>> When I implemented all the changes (and also set it up
so it's
> > >>>>>>>>>> reading the beta6 version), I used this command:
> > >>>>>>>>>>
> > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > >>>>>>>>>>
> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
-v
> 4
> > >>>>>>>>>>
> > >>>>>>>>>> But then I got this error:
> > >>>>>>>>>>
> > >>>>>>>>>> DEBUG 2:
> > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > >>>>>>>>>> records found for 'OZMAX1/L1'
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> ERROR :
> > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> > data
> > >>>>>>>>>> found! Exiting...
> > >>>>>>>>>>
> > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the
> file
> > >>>>>>>>>> in this case reads:
> > >>>>>>>>>>
> > >>>>>>>>>> 14 (t14a2)
> > >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>
> > wgrib2 oz1hr_DAY1
> > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > >>>>>>>>>> 15 (t14a2)
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>>>>>>>>
> > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
suggested
> > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe it
> > needs to be
> > >>>>>>>>>> the time stamp of the first day?
> > >>>>>>>>>>
> > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for any thoughts.
> > >>>>>>>>>>
> > >>>>>>>>>> Perry
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Perry,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm confused. In the previous air quality vx we did,
we used
> > >>>>>>>>>>> observation files named:
> > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > >>>>>>>>>>>
> > >>>>>>>>>>> And those files don't exist in the July data you just
sent.
> > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm not sure if you're actually verifying against the
BUFR or
> > >>>>>>>>>>> PREPBUFR.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > >>>>>>>>>>> settings...
> > >>>>>>>>>>>
> > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files -
> > >>>>>>>>>>> thecurrent day and the previous day...
> > >>>>>>>>>>> pb2nc \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > >>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>
> > >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
like
> > this:
> > >>>>>>>>>>> time_summary = {
> > >>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>> end = "040000";
> > >>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > >>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > >>>>>>>>>>> type = [ "max", "mean" ];
> > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at
> least
> > >>>>>>>>>>> 75% of expect obs are present
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station over
> > the
> > >>>>>>>>>>> defined time period. They're identified in the output
by the
> > message type:
> > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > >>>>>>>>>>>
> > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead
> of
> > specifying
> > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient to
> > specify the
> > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> file,
> > I'd suggest
> > >>>>>>>>>>> using environment variables:
> > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > >>>>>>>>>>>
> > >>>>>>>>>>> fcst = {
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>> obs = {
> > >>>>>>>>>>> convert(x) = x * 10^9;
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Sounds good, thanks!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> John
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> > Affiliate <
> > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and focus
> > on
> > >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
over to
> > Theia so we'll
> > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> into
> > problems, I'll
> > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
max/avg
> > files).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for bias
> > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
you
> may
> > want to provide
> > >>>>>>>>>>>>>> him para5 runs.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John to
> do
> > >>>>>>>>>>>>>> some testing some production files. Do you think I
should
> > use parallels
> > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
creating
> a
> > MET+ use case
> > >>>>>>>>>>>>>> for AQ?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > should
> > >>>>>>>>>>>>>> be OK for now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
We
> > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels but
> > management did not
> > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> > >>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > parallel CMAQ data
> > >>>>>>>>>>>>>>> rather than production CMAQ.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions of
> > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can also
> > clarify this as
> > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over the
> > current wgrib2?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use the
> > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping also
> > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
did
> not
> > want to risk any
> > >>>>>>>>>>>>>>>> further delays so we will have to do so with the
FYQ4
> > implementation.
> > >>>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think I
> > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago, but
> > I don't know if I
> > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think the
> > parallel CMAQ they
> > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> > But he would have
> > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> > this.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
talks
> > >>>>>>>>>>>>>>>> about?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Perry,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute
> the
> > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
hour
> > ozone concentrations.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > pb2nc_output.nc
> > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Using the attached config files.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
FULL,
> > for interpolation
> > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
this
> > "max"
> > >>>>>>>>>>>>>>>>> file:
> > >>>>>>>>>>>>>>>>> wgrib2
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-
40 hour
> > >>>>>>>>>>>>>>>>> ave fcst:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
> revealed
> > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17 and
> > 41, respectively.
> > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same file,
> > we
> > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one to
> > use. Do this by
> > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > >>>>>>>>>>>>>>>>> fcst = {
> > >>>>>>>>>>>>>>>>> field = [
> > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> > >>>>>>>>>>>>>>>>> "41";' }
> > >>>>>>>>>>>>>>>>> ];
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off by
> 1
> > >>>>>>>>>>>>>>>>> hour.
> > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
time =
> > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value from
> > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while the
> > obs are valid at 04Z.
> > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config file
> > by
> > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > >>>>>>>>>>>>>>>>> obs_window = {
> > >>>>>>>>>>>>>>>>> beg = -3600;
> > >>>>>>>>>>>>>>>>> end = -3600;
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> So what're the next steps?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> found
> > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> > >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-hour
> > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output to
> > an ascii file using
> > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count up
> the
> > number of regular
> > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
| wc
> -l
> > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
| sort
> > -u
> > >>>>>>>>>>>>>>>>>> | wc -l
> > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
wc -l
> > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
sort
> -u |
> > >>>>>>>>>>>>>>>>>> wc -l
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
all of
> > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
> meaning
> > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> duplicates.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is named
> > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
> 2FB31EEC
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the last
> > two
> > >>>>>>>>>>>>>>>>>> lines.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates are
> > >>>>>>>>>>>>>>>>>> showing up?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Cheers.
> > >>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
we'd
> have
> > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across
2
> > files. Do we have existing
> > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
compute
> > a 24-hour max
> > >>>>>>>>>>>>>>>>>>> temperature or something?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> sec).
> > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the new
> > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
<hsoh at ucar.edu
> >
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content
of
> > obs_hid
> > >>>>>>>>>>>>>>>>>>>>> variable).
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
(test
> > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
added
> > one
> > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > check the unit test today.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> nightly
> > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> > covered? If not, should I add
> > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
data.
> > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
> Gotway <
> > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> PB2NC.
> > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> > maximum values of COPO from
> > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump to
> > ascii
> > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string show
> > up in the output.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > actually
> > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
Rscript
> > dumping the data.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> Gotway
> > <
> > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
57246
> > observations
> > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00. There
> > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
I'm
> > working on this.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > Gotway
> > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
take a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC.
I
> > copied the data over to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> \
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
> maximums
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end =
0; } //
> > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > expect. The hdr_typ_table
> > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > message type, but as you pointed
> > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> using 2
> > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> > cover the desired 04Z to 04Z
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR file
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
hour
> > maximum of obs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable for
> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> NetCDF
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > "AIRNOW_MAX_240000".
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
generated.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
than
> obs
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
output:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> currently)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> exists
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The first
> > header index for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1 -d"
" |
> sort
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota
in
> here:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180622
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > svn-met-dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
them
> > should actually be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
lookup
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Fri Apr 26 10:41:06 2019
Thanks Perry,
We have your files and will have a look...
David
On Fri, Apr 26, 2019 at 10:26 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, David,
>
> I put all you need in the directory /incoming/irap/met_help/pshafran
on
> your ftp server.
>
> Let me know if you require more items or need some explanation of
what I
> have here.
>
> Thanks!
>
> Perry
>
> On Fri, Apr 26, 2019 at 12:08 PM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Perry -
> >
> > I'm also forwarding this ticket to Howard,
> > I think John is quite busy with the MET 8.1 release so it might be
later
> > next week before he gets a chance to look at this.
> >
> > If you want to upload your data and config files to our ftp server
we can
> > replicate and debug the error here.
> > I can have a first look this afternoon.
> > You can create a subdir at
> > ftp.rap.ucar.edu://incoming/irap/met_help
> >
> > thanks,
> > David
> >
> > On Thu, Apr 25, 2019 at 11:17 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Apr 25 11:16:42 2019: Request 89921 was acted upon.
> > > Transaction: Ticket created by perry.shafran at noaa.gov
> > > Queue: met_help
> > > Subject: Re: PB2NC time summary questions
> > > Owner: Nobody
> > > Requestors: perry.shafran at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> >
> > >
> > >
> > > Hi, everyone,
> > >
> > > I am attaching this to this long thread regarding the daily max
of
> ozone
> > > verification, so we remember what we were doing back then.
> > >
> > > I am now running this again using MET 8.0, and something seems
to not
> be
> > > working when I run point_stat. I get this error when running a
daily
> max
> > > of ozone:
> > >
> > > ERROR :
> > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > ERROR :
> > >
> > > I am using my own scripts to run this using MET, so this is not
using
> > > METplus.
> > >
> > > You can find a log file of this here:
> > >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > It
> > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> fine,
> > I
> > > believe.
> > >
> > > The script I am using to run this is here:
> > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > >
> > > The prepbufr files are:
> > >
> > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > >
> > > The model file is here:
> > >
> > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > >
> > >
> > > What could be causing this error?
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > >
> > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Great, glad that did the trick.
> > > >
> > > > John
> > > >
> > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate
<
> > > > perry.shafran at noaa.gov> wrote:
> > > >
> > > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-
hr avg.
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Perry
> > > >>
> > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway <
> johnhg at ucar.edu>
> > > >> wrote:
> > > >>
> > > >>> Perry,
> > > >>>
> > > >>> I think the problem is in your setting of
> > valid_time="${CUR_VX_TIME}";
> > > >>>
> > > >>> Here's your wgrib2 output:
> > > >>> wgrib2 oz_DAY1
> > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > >>>
> > > >>> The "d=2018081112" tells you the model initialization time.
The
> > valid
> > > >>> time = init time + forecast time. So the OZMAX1 data is
valid 16
> > hours
> > > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> > wgrib2
> > > >>> command using the "-vt" argument to list the verification
time.
> > > >>>
> > > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > valid
> > > >>> at the time you're requesting.
> > > >>>
> > > >>> John
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate <
> > > >>> perry.shafran at noaa.gov> wrote:
> > > >>>
> > > >>>> Hi, John,
> > > >>>>
> > > >>>> I haven't been able to work on this for awhile, but now I'm
> working
> > on
> > > >>>> it again but seem to be running into issues.
> > > >>>>
> > > >>>> I have a file that has both the daily max for 1-hr ozone
and the
> > daily
> > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > >>>>
> > > >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > > >>>> wgrib2 oz_DAY1
> > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > >>>> 10 (t10a2)
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > >>>>
> > > >>>> As you can see they have different forecast times. But
when I run
> > > >>>> point_stat, I get this output:
> > > >>>>
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2:
> > > >>>>
> > >
> >
>
--------------------------------------------------------------------------------
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > >>>> WARNING:
> > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> records
> > > >>>> found for 'OZMAX8/L1'
> > > >>>> WARNING:
> > > >>>> WARNING:
> > > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> OZMAX8/L1
> > > >>>> found in file: oz_DAY1
> > > >>>> WARNING:
> > > >>>>
> > > >>>> Here is the relevant portion of the point_stat config file:
> > > >>>>
> > > >>>> fcst = {
> > > >>>> field = [
> > > >>>> { name = "OZMAX1"; level = "L1";
> valid_time="${CUR_VX_TIME}";
> > },
> > > >>>> { name = "OZMAX8"; level = "L1";
> valid_time="${CUR_VX_TIME}";
> > }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> obs = {
> > > >>>> convert(x) = x * 10^9;
> > > >>>> field = [
> > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; },
> > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> I don't remember if I had this working previously for both
the 1hr
> > and
> > > >>>> the 8hr ozone, but apparently there is some sort of issue
for the
> > 8hr
> > > >>>> ozone. Have any thoughts?
> > > >>>>
> > > >>>> Thanks!
> > > >>>>
> > > >>>> Perry
> > > >>>>
> > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate <
> > > >>>> perry.shafran at noaa.gov> wrote:
> > > >>>>
> > > >>>>> I chose option 2, and that seems to work - I have a .stat
file
> with
> > > >>>>> output. Now I'll need to do some work here to see if the
output
> > > looks OK,
> > > >>>>> and to configure the verification so it verifies OZMAX1
and
> OZMAX8.
> > > >>>>>
> > > >>>>> I also wonder if this would be easily modified for PM.
> > > >>>>>
> > > >>>>> Perry
> > > >>>>>
> > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Perry,
> > > >>>>>>
> > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries that
> > are
> > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that are
> > > purportedly
> > > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > > rejected due
> > > >>>>>> to the valid time.
> > > >>>>>>
> > > >>>>>> It's up to you guys to decide what the "correct" way of
handling
> > > this
> > > >>>>>> is... but here are a couple of options:
> > > >>>>>>
> > > >>>>>> (1) You could use the existing setup and address this
logic in
> > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> from
> > 0
> > > to:
> > > >>>>>>
> > > >>>>>> obs_window = {
> > > >>>>>> beg = 3600;
> > > >>>>>> end = 3600;
> > > >>>>>> }
> > > >>>>>> This tells Point-Stat to match observations which are
exactly
> > 1-hour
> > > >>>>>> prior to the valid time of the forecast.
> > > >>>>>>
> > > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z to
> > 05Z
> > > >>>>>> instead and then leave the Point-Stat obs_window setting
at 0:
> > > >>>>>>
> > > >>>>>> time_summary = {
> > > >>>>>> flag = TRUE;
> > > >>>>>> beg = "050000";
> > > >>>>>> end = "050000";
> > > >>>>>> ...
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> John
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate
> <
> > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>
> > > >>>>>>> Oops, spoke too soon. There *is* a .stat output file,
but it's
> > > >>>>>>> empty. Here's the output from point_stat:
> > > >>>>>>>
> > > >>>>>>> DEBUG 4:
> > > >>>>>>> DEBUG 4: Data plane information:
> > > >>>>>>> DEBUG 4: plane min: 13.86
> > > >>>>>>> DEBUG 4: plane max: 130.82
> > > >>>>>>> DEBUG 4: scan mode: 64
> > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > >>>>>>> DEBUG 4: lead time: 170000
> > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > type
> > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > > NEAREST(1),
> > > >>>>>>> using 0 pairs.
> > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > >>>>>>>
> > > >>>>>>> Looks like most rejections were due to valid time.
> > > >>>>>>>
> > > >>>>>>> Have any thoughts about this one?
> > > >>>>>>>
> > > >>>>>>> Thanks!
> > > >>>>>>>
> > > >>>>>>> Perry
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> Affiliate <
> > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi, John,
> > > >>>>>>>>
> > > >>>>>>>> Looks like the changing of the time stamp from _04 to
_05 did
> > the
> > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > >>>>>>>>
> > > >>>>>>>> Thanks!
> > > >>>>>>>>
> > > >>>>>>>> Perry
> > > >>>>>>>>
> > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Perry,
> > > >>>>>>>>>
> > > >>>>>>>>> I agree, there's likely some issue with requesting the
> > timestamp
> > > >>>>>>>>> of the data from that file. I think switching the
> verification
> > > time to
> > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > >>>>>>>>>
> > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > times
> > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2 and
> > > MET commands
> > > >>>>>>>>> to illustrate.
> > > >>>>>>>>>
> > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast time
> > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> data
> > > file you
> > > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> so I
> > > have that
> > > >>>>>>>>> funky forecast hour:
> > > >>>>>>>>>
> > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > >>>>>>>>> fcst:vt=2018070505
> > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > >>>>>>>>>
> > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level
> > (-v
> > > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > > reading:
> > > >>>>>>>>>
> > > >>>>>>>>> # Read the first record
> > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>> ...
> > > >>>>>>>>> # Read the second record
> > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>>
> > > >>>>>>>>> Hope that helps.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> John
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> Affiliate
> > <
> > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi, John,
> > > >>>>>>>>>>
> > > >>>>>>>>>> When I implemented all the changes (and also set it
up so
> it's
> > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > >>>>>>>>>>
> > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > >>>>>>>>>>
> > >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> -v
> > 4
> > > >>>>>>>>>>
> > > >>>>>>>>>> But then I got this error:
> > > >>>>>>>>>>
> > > >>>>>>>>>> DEBUG 2:
> > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> ERROR :
> > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> > > data
> > > >>>>>>>>>> found! Exiting...
> > > >>>>>>>>>>
> > > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output
for the
> > file
> > > >>>>>>>>>> in this case reads:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 14 (t14a2)
> > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > wgrib2 oz1hr_DAY1
> > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>>>>>>>>> 15 (t14a2)
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>>>>>>>>
> > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> suggested
> > > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> it
> > > needs to be
> > > >>>>>>>>>> the time stamp of the first day?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for any thoughts.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Perry
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Perry,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did, we
> used
> > > >>>>>>>>>>> observation files named:
> > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> And those files don't exist in the July data you
just sent.
> > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm not sure if you're actually verifying against
the BUFR
> or
> > > >>>>>>>>>>> PREPBUFR.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > > >>>>>>>>>>> settings...
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files
> -
> > > >>>>>>>>>>> thecurrent day and the previous day...
> > > >>>>>>>>>>> pb2nc \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > >>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary like
> > > this:
> > > >>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > >>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries
when at
> > least
> > > >>>>>>>>>>> 75% of expect obs are present
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> over
> > > the
> > > >>>>>>>>>>> defined time period. They're identified in the
output by
> the
> > > message type:
> > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > > >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead
> > of
> > > specifying
> > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> to
> > > specify the
> > > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > file,
> > > I'd suggest
> > > >>>>>>>>>>> using environment variables:
> > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> fcst = {
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>> obs = {
> > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Sounds good, thanks!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> John
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > Affiliate <
> > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and
> focus
> > > on
> > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files over
> to
> > > Theia so we'll
> > > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> > into
> > > problems, I'll
> > > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-
hr
> max/avg
> > > files).
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for
> bias
> > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe, you
> > may
> > > want to provide
> > > >>>>>>>>>>>>>> him para5 runs.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> to
> > do
> > > >>>>>>>>>>>>>> some testing some production files. Do you think
I
> should
> > > use parallels
> > > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> creating
> > a
> > > MET+ use case
> > > >>>>>>>>>>>>>> for AQ?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > > should
> > > >>>>>>>>>>>>>> be OK for now.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels. We
> > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels but
> > > management did not
> > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > >>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > > parallel CMAQ data
> > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions
> of
> > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can
> also
> > > clarify this as
> > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over the
> > > current wgrib2?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use
> the
> > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> also
> > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management did
> > not
> > > want to risk any
> > > >>>>>>>>>>>>>>>> further delays so we will have to do so with
the FYQ4
> > > implementation.
> > > >>>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think
> I
> > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> but
> > > I don't know if I
> > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> the
> > > parallel CMAQ they
> > > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> > > But he would have
> > > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> > > this.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues
John
> talks
> > > >>>>>>>>>>>>>>>> about?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute
> > the
> > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and
8-hour
> > > ozone concentrations.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region FULL,
> > > for interpolation
> > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents
of this
> > > "max"
> > > >>>>>>>>>>>>>>>>> file:
> > > >>>>>>>>>>>>>>>>> wgrib2
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
level:17-40
> hour
> > > >>>>>>>>>>>>>>>>> ave fcst:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > revealed
> > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours 17
> and
> > > 41, respectively.
> > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> file,
> > > we
> > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one to
> > > use. Do this by
> > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > >>>>>>>>>>>>>>>>> fcst = {
> > > >>>>>>>>>>>>>>>>> field = [
> > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> > > >>>>>>>>>>>>>>>>> "41";' }
> > > >>>>>>>>>>>>>>>>> ];
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps
are off
> by
> > 1
> > > >>>>>>>>>>>>>>>>> hour.
> > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead time
> =
> > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> from
> > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> the
> > > obs are valid at 04Z.
> > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> file
> > > by
> > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > > >>>>>>>>>>>>>>>>> obs_window = {
> > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > >>>>>>>>>>>>>>>>> end = -3600;
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> > found
> > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-hour
> > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> to
> > > an ascii file using
> > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count up
> > the
> > > number of regular
> > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt | wc
> > -l
> > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> sort
> > > -u
> > > >>>>>>>>>>>>>>>>>> | wc -l
> > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt
| wc -l
> > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt
| sort
> > -u |
> > > >>>>>>>>>>>>>>>>>> wc -l
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning
that all
> of
> > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > meaning
> > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > duplicates.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is
> named
> > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > 2FB31EEC
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the
> last
> > > two
> > > >>>>>>>>>>>>>>>>>> lines.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> are
> > > >>>>>>>>>>>>>>>>>> showing up?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu
> >
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > >>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason
why we'd
> > have
> > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > files. Do we have existing
> > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> compute
> > > a 24-hour max
> > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > sec).
> > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the
> new
> > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh <
> hsoh at ucar.edu
> > >
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content of
> > > obs_hid
> > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> (test
> > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs.
I
> added
> > > one
> > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > > check the unit test today.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> > nightly
> > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > covered? If not, should I add
> > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh
<
> > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> data.
> > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > Gotway <
> > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> > PB2NC.
> > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute
24-hour
> > > maximum values of COPO from
> > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump to
> > > ascii
> > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> show
> > > up in the output.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > > actually
> > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in
the
> Rscript
> > > dumping the data.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > Gotway
> > > <
> > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00
and
> 57246
> > > observations
> > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> There
> > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data. I'm
> > > working on this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > > Gotway
> > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> take a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC. I
> > > copied the data over to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > maximums
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end
= 0; }
> //
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > > expect. The hdr_typ_table
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > > message type, but as you pointed
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> > using 2
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of
them to
> > > cover the desired 04Z to 04Z
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> file
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a
24 hour
> > > maximum of obs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh
<
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable for
> > me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> > NetCDF
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > "AIRNOW_MAX_240000".
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> generated.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers than
> > obs
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> output:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> > currently)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0
to
> 3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > exists
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The
> first
> > > header index for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1
-d" " |
> > sort
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > > wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota in
> > here:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > svn-met-
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some
of them
> > > should actually be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced
by this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> lookup
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Thu May 02 15:35:10 2019
Hi Perry -
An update on where we are on this issue:
Using the prepda.2019042312.nc file you provided we indeed reproduce
the point_stat error.
The point_stat error is due to only a single entry the valid time
table,
e.g. from ncdump -v hdr_vld_table
hdr_vld_table =
"20190423_050000",
while the hdr_vld indices in that file are hdr_vld = 0, 33, 33, ...
which are all out of range except the first.
If I run pb2nc here with your PREPBUFR data and your PB2NCConfig_aqm
config file, however,
I generate a different prepda.2019042312.nc file that runs through to
completion with point_stat.
Its valid time table:
hdr_vld_table =
"20190423_030000",
"20190423_010000",
"20190423_020000",
"20190423_230000",
"20190423_150000",
"20190423_090000",
"20190423_070000",
"20190423_060000",
"20190423_210000",
"20190423_050000",
"20190423_170000",
"20190423_100000",
"20190423_200000",
"20190423_140000",
"20190423_130000",
"20190423_160000",
"20190423_040000",
"20190423_180000",
"20190423_080000",
"20190423_120000",
"20190423_110000",
"20190423_220000",
"20190423_190000",
"20190424_000000" ;
}
So it seems pb2nc is running correctly here, but not on your side.
Let me discuss with John the best way to proceed ...
thanks,
David
On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> Hi, everyone,
>
> I am attaching this to this long thread regarding the daily max of
> ozone
> verification, so we remember what we were doing back then.
>
> I am now running this again using MET 8.0, and something seems to
not
> be
> working when I run point_stat. I get this error when running a
daily
> max
> of ozone:
>
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> I am using my own scripts to run this using MET, so this is not
using
> METplus.
>
> You can find a log file of this here:
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> It
> runs pb2nc and point_stat for max ozone. The pb2nc code is working
> fine, I
> believe.
>
> The script I am using to run this is here:
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
>
> The prepbufr files are:
>
> /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
>
> The model file is here:
>
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
>
>
> What could be causing this error?
>
> Thanks!
>
> Perry
>
>
> On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Great, glad that did the trick.
> >
> > John
> >
> > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > perry.shafran at noaa.gov> wrote:
> >
> >> Well this worked! And Jeff confirmed to me that the 8-hr average
> >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
avg.
> >>
> >> Thanks!
> >>
> >> Perry
> >>
> >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> >> <johnhg at ucar.edu>
> >> wrote:
> >>
> >>> Perry,
> >>>
> >>> I think the problem is in your setting of
> >>> valid_time="${CUR_VX_TIME}";
> >>>
> >>> Here's your wgrib2 output:
> >>> wgrib2 oz_DAY1
> >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> >>>
> >>> The "d=2018081112" tells you the model initialization time. The
> >>> valid
> >>> time = init time + forecast time. So the OZMAX1 data is valid
16
> >>> hours
> >>> later while the OZMAX8 data is valid 22 hours later. Rerun that
> >>> wgrib2
> >>> command using the "-vt" argument to list the verification time.
> >>>
> >>> Point-Stat isn't finding any OZMAX8 data because that data isn't
> >>> valid
> >>> at the time you're requesting.
> >>>
> >>> John
> >>>
> >>>
> >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate <
> >>> perry.shafran at noaa.gov> wrote:
> >>>
> >>>> Hi, John,
> >>>>
> >>>> I haven't been able to work on this for awhile, but now I'm
> >>>> working on
> >>>> it again but seem to be running into issues.
> >>>>
> >>>> I have a file that has both the daily max for 1-hr ozone and
the
> >>>> daily
> >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> >>>>
> >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>> >
> >>>> wgrib2 oz_DAY1
> >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
> >>>> 10 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>> >
> >>>>
> >>>> As you can see they have different forecast times. But when I
run
> >>>> point_stat, I get this output:
> >>>>
> >>>> DEBUG 2:
> >>>> DEBUG 2:
> >>>>
--------------------------------------------------------------------------------
> >>>> DEBUG 2:
> >>>> DEBUG 2: Reading data for OZMAX8/L1.
> >>>> WARNING:
> >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> >>>> records
> >>>> found for 'OZMAX8/L1'
> >>>> WARNING:
> >>>> WARNING:
> >>>> WARNING: process_fcst_climo_files() -> no fields matching
> >>>> OZMAX8/L1
> >>>> found in file: oz_DAY1
> >>>> WARNING:
> >>>>
> >>>> Here is the relevant portion of the point_stat config file:
> >>>>
> >>>> fcst = {
> >>>> field = [
> >>>> { name = "OZMAX1"; level = "L1";
> >>>> valid_time="${CUR_VX_TIME}"; },
> >>>> { name = "OZMAX8"; level = "L1";
> >>>> valid_time="${CUR_VX_TIME}"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> obs = {
> >>>> convert(x) = x * 10^9;
> >>>> field = [
> >>>> { name = "COPO"; level = "A1"; message_type =
> >>>> "AIRNOW_MAX_230000"; },
> >>>> { name = "COPO"; level = "A8"; message_type =
> >>>> "AIRNOW_MAX_230000"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> I don't remember if I had this working previously for both the
1hr
> >>>> and
> >>>> the 8hr ozone, but apparently there is some sort of issue for
the
> >>>> 8hr
> >>>> ozone. Have any thoughts?
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Perry
> >>>>
> >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA Affiliate
<
> >>>> perry.shafran at noaa.gov> wrote:
> >>>>
> >>>>> I chose option 2, and that seems to work - I have a .stat file
> >>>>> with
> >>>>> output. Now I'll need to do some work here to see if the
output
> >>>>> looks OK,
> >>>>> and to configure the verification so it verifies OZMAX1 and
> >>>>> OZMAX8.
> >>>>>
> >>>>> I also wonder if this would be easily modified for PM.
> >>>>>
> >>>>> Perry
> >>>>>
> >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> >>>>> <johnhg at ucar.edu>
> >>>>> wrote:
> >>>>>
> >>>>>> Perry,
> >>>>>>
> >>>>>> Yep, you've configured/run PB2NC to compute time summaries
that
> >>>>>> are
> >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> >>>>>> purportedly
> >>>>>> valid at 05Z. And that's the reason why you have so many obs
> >>>>>> rejected due
> >>>>>> to the valid time.
> >>>>>>
> >>>>>> It's up to you guys to decide what the "correct" way of
handling
> >>>>>> this
> >>>>>> is... but here are a couple of options:
> >>>>>>
> >>>>>> (1) You could use the existing setup and address this logic
in
> >>>>>> Point-Stat config file. Change the matching time obs_window
> >>>>>> from 0 to:
> >>>>>>
> >>>>>> obs_window = {
> >>>>>> beg = 3600;
> >>>>>> end = 3600;
> >>>>>> }
> >>>>>> This tells Point-Stat to match observations which are exactly
1-
> >>>>>> hour
> >>>>>> prior to the valid time of the forecast.
> >>>>>>
> >>>>>> (2) You could rerun PB2NC to compute a time summary from 05Z
to
> >>>>>> 05Z
> >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> >>>>>>
> >>>>>> time_summary = {
> >>>>>> flag = TRUE;
> >>>>>> beg = "050000";
> >>>>>> end = "050000";
> >>>>>> ...
> >>>>>>
> >>>>>> Thanks,
> >>>>>> John
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate
> >>>>>> <
> >>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>
> >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
it's
> >>>>>>> empty. Here's the output from point_stat:
> >>>>>>>
> >>>>>>> DEBUG 4:
> >>>>>>> DEBUG 4: Data plane information:
> >>>>>>> DEBUG 4: plane min: 13.86
> >>>>>>> DEBUG 4: plane max: 130.82
> >>>>>>> DEBUG 4: scan mode: 64
> >>>>>>> DEBUG 4: valid time: 20180709_050000
> >>>>>>> DEBUG 4: lead time: 170000
> >>>>>>> DEBUG 4: init time: 20180708_120000
> >>>>>>> DEBUG 4: bitmap flag: 255
> >>>>>>>
> >>>>>>>
> >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> >>>>>>> type
> >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> >>>>>>> NEAREST(1),
> >>>>>>> using 0 pairs.
> >>>>>>> DEBUG 3: Number of matched pairs = 0
> >>>>>>> DEBUG 3: Observations processed = 124637
> >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> >>>>>>> DEBUG 3: Rejected: obs type = 0
> >>>>>>> DEBUG 3: Rejected: valid time = 122154
> >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> >>>>>>> DEBUG 3: Rejected: off the grid = 70
> >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> >>>>>>> DEBUG 3: Rejected: quality marker = 0
> >>>>>>> DEBUG 3: Rejected: message type = 1189
> >>>>>>> DEBUG 3: Rejected: masking region = 0
> >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> >>>>>>> DEBUG 3: Rejected: duplicates = 0
> >>>>>>>
> >>>>>>> Looks like most rejections were due to valid time.
> >>>>>>>
> >>>>>>> Have any thoughts about this one?
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> Perry
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
Affiliate
> >>>>>>> <
> >>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>
> >>>>>>>> Hi, John,
> >>>>>>>>
> >>>>>>>> Looks like the changing of the time stamp from _04 to _05
did
> >>>>>>>> the
> >>>>>>>> trick as point_stat ran without an issue with that change.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Perry
> >>>>>>>>
> >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> >>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>
> >>>>>>>>> Perry,
> >>>>>>>>>
> >>>>>>>>> I agree, there's likely some issue with requesting the
> >>>>>>>>> timestamp
> >>>>>>>>> of the data from that file. I think switching the
> >>>>>>>>> verification time to
> >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> >>>>>>>>>
> >>>>>>>>> Looks like wgrib2 and MET both agree on the initialization
> >>>>>>>>> times
> >>>>>>>>> and valid times for this data. Here's some useful wgrib2
and
> >>>>>>>>> MET commands
> >>>>>>>>> to illustrate.
> >>>>>>>>>
> >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
time
> >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a sample
> >>>>>>>>> data file you
> >>>>>>>>> sent to me. But remember I'm still using the "old" wgrib2
so
> >>>>>>>>> I have that
> >>>>>>>>> funky forecast hour:
> >>>>>>>>>
> >>>>>>>>> wgrib2 -t -ftime -vt
> >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> >>>>>>>>> fcst:vt=2018070505
> >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
> >>>>>>>>>
> >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level
> >>>>>>>>> (-v
> >>>>>>>>> 4) to have MET tell you the timing info of the data it is
> >>>>>>>>> reading:
> >>>>>>>>>
> >>>>>>>>> # Read the first record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> >>>>>>>>> DEBUG 4: lead time: 230000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>> ...
> >>>>>>>>> # Read the second record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> >>>>>>>>> DEBUG 4: lead time: 470000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>>
> >>>>>>>>> Hope that helps.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> John
> >>>>>>>>>
> >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> >>>>>>>>> Affiliate <
> >>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi, John,
> >>>>>>>>>>
> >>>>>>>>>> When I implemented all the changes (and also set it up so
> >>>>>>>>>> it's
> >>>>>>>>>> reading the beta6 version), I used this command:
> >>>>>>>>>>
> >>>>>>>>>> CUR_VX_TIME=20180709_04
> >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> >>>>>>>>>>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> >>>>>>>>>> -v 4
> >>>>>>>>>>
> >>>>>>>>>> But then I got this error:
> >>>>>>>>>>
> >>>>>>>>>> DEBUG 2:
> >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> >>>>>>>>>> records found for 'OZMAX1/L1'
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields matching
> >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> >>>>>>>>>> WARNING:
> >>>>>>>>>> ERROR :
> >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> >>>>>>>>>> data
> >>>>>>>>>> found! Exiting...
> >>>>>>>>>>
> >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the
> >>>>>>>>>> file
> >>>>>>>>>> in this case reads:
> >>>>>>>>>>
> >>>>>>>>>> 14 (t14a2)
> >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> >>>>>>>>>> wgrib2 oz1hr_DAY1
> >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> >>>>>>>>>> 15 (t14a2)
> >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>>>>>>>>
> >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> >>>>>>>>>> suggested
> >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> >>>>>>>>>> it needs to be
> >>>>>>>>>> the time stamp of the first day?
> >>>>>>>>>>
> >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for any thoughts.
> >>>>>>>>>>
> >>>>>>>>>> Perry
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> >>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Perry,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm confused. In the previous air quality vx we did, we
> >>>>>>>>>>> used
> >>>>>>>>>>> observation files named:
> >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> >>>>>>>>>>>
> >>>>>>>>>>> And those files don't exist in the July data you just
sent.
> >>>>>>>>>>> Instead, it's BUFR files with names like:
> >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure if you're actually verifying against the
BUFR
> >>>>>>>>>>> or
> >>>>>>>>>>> PREPBUFR.
> >>>>>>>>>>>
> >>>>>>>>>>> Let me just tell you the key points for the config file
> >>>>>>>>>>> settings...
> >>>>>>>>>>>
> >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files
> >>>>>>>>>>> -
> >>>>>>>>>>> thecurrent day and the previous day...
> >>>>>>>>>>> pb2nc \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> >>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>
> >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
like
> >>>>>>>>>>> this:
> >>>>>>>>>>> time_summary = {
> >>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>> beg = "040000";
> >>>>>>>>>>> end = "040000";
> >>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> >>>>>>>>>>> 23-hours (05Z to 04Z)
> >>>>>>>>>>> grib_code = [];
> >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> >>>>>>>>>>> type = [ "max", "mean" ];
> >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at
> >>>>>>>>>>> least
> >>>>>>>>>>> 75% of expect obs are present
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>> That'll compute the MAX and MEAN values at each station
> >>>>>>>>>>> over the
> >>>>>>>>>>> defined time period. They're identified in the output
by
> >>>>>>>>>>> the message type:
> >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> >>>>>>>>>>>
> >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead
> >>>>>>>>>>> of specifying
> >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> >>>>>>>>>>> to specify the
> >>>>>>>>>>> valid time. So that you don't have to modify the config
> >>>>>>>>>>> file, I'd suggest
> >>>>>>>>>>> using environment variables:
> >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> >>>>>>>>>>>
> >>>>>>>>>>> fcst = {
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>> obs = {
> >>>>>>>>>>> convert(x) = x * 10^9;
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> >>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Sounds good, thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> John
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> >>>>>>>>>>>> Affiliate <
> >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids and
> >>>>>>>>>>>>> focus on
> >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
over
> >>>>>>>>>>>>> to Theia so we'll
> >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> >>>>>>>>>>>>> into problems, I'll
> >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
> >>>>>>>>>>>>> max/avg files).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not for
> >>>>>>>>>>>>>> bias
> >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
you
> >>>>>>>>>>>>>> may want to provide
> >>>>>>>>>>>>>> him para5 runs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> >>>>>>>>>>>>>> to do
> >>>>>>>>>>>>>> some testing some production files. Do you think I
> >>>>>>>>>>>>>> should use parallels
> >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> >>>>>>>>>>>>>> creating a MET+ use case
> >>>>>>>>>>>>>> for AQ?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> >>>>>>>>>>>>>> should
> >>>>>>>>>>>>>> be OK for now.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
We
> >>>>>>>>>>>>>>> wanted to update the error in daily max hour labels
but
> >>>>>>>>>>>>>>> management did not
> >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> >>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA Affiliate
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John over
> >>>>>>>>>>>>>>> parallel CMAQ data
> >>>>>>>>>>>>>>> rather than production CMAQ.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they can
> >>>>>>>>>>>>>>> also clarify this as
> >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does over
the
> >>>>>>>>>>>>>>> current wgrib2?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to use
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> latest parallel run output (para5).
> >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall Jianping
> >>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
did
> >>>>>>>>>>>>>>>> not want to risk any
> >>>>>>>>>>>>>>>> further delays so we will have to do so with the
FYQ4
> >>>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA Affiliate
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think
> >>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> >>>>>>>>>>>>>>>> but I don't know if I
> >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I think
> >>>>>>>>>>>>>>>> the parallel CMAQ they
> >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> >>>>>>>>>>>>>>>> But he would have
> >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> >>>>>>>>>>>>>>>> this.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
> >>>>>>>>>>>>>>>> talks
> >>>>>>>>>>>>>>>> about?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley Gotway
<
> >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Perry,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
hour
> >>>>>>>>>>>>>>>>> ozone concentrations.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>> pb2nc_output.nc
> >>>>>>>>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
\
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
\
> >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Using the attached config files.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
FULL,
> >>>>>>>>>>>>>>>>> for interpolation
> >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But I noticed some oddities:
> >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
this
> >>>>>>>>>>>>>>>>> "max"
> >>>>>>>>>>>>>>>>> file:
> >>>>>>>>>>>>>>>>> wgrib2
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-40
> >>>>>>>>>>>>>>>>> hour
> >>>>>>>>>>>>>>>>> ave fcst:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
> >>>>>>>>>>>>>>>>> revealed
> >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17
> >>>>>>>>>>>>>>>>> and 41, respectively.
> >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the same
> >>>>>>>>>>>>>>>>> file, we
> >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which one
to
> >>>>>>>>>>>>>>>>> use. Do this by
> >>>>>>>>>>>>>>>>> defining the "lead_time":
> >>>>>>>>>>>>>>>>> fcst = {
> >>>>>>>>>>>>>>>>> field = [
> >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> >>>>>>>>>>>>>>>>> "41";' }
> >>>>>>>>>>>>>>>>> ];
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off
> >>>>>>>>>>>>>>>>> by 1
> >>>>>>>>>>>>>>>>> hour.
> >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
time
> >>>>>>>>>>>>>>>>> =
> >>>>>>>>>>>>>>>>> valid time of 2018052805.
> >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max value
> >>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z while
> >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat config
> >>>>>>>>>>>>>>>>> file by
> >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> >>>>>>>>>>>>>>>>> obs_window = {
> >>>>>>>>>>>>>>>>> beg = -3600;
> >>>>>>>>>>>>>>>>> end = -3600;
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So what're the next steps?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We're making progress on the time summaries, but
I
> >>>>>>>>>>>>>>>>>> found
> >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute 24-
hour
> >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> >>>>>>>>>>>>>>>>>> to an ascii file using
> >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to count
up
> >>>>>>>>>>>>>>>>>> the number of regular
> >>>>>>>>>>>>>>>>>> and maximum obs values:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
wc
> >>>>>>>>>>>>>>>>>> -l
> >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
> >>>>>>>>>>>>>>>>>> sort -u
> >>>>>>>>>>>>>>>>>> | wc -l
> >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt | wc
-l
> >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
sort
> >>>>>>>>>>>>>>>>>> -u |
> >>>>>>>>>>>>>>>>>> wc -l
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
all
> >>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
> >>>>>>>>>>>>>>>>>> meaning
> >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> >>>>>>>>>>>>>>>>>> duplicates.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> For example, the last station ID in that file is
> >>>>>>>>>>>>>>>>>> named
> >>>>>>>>>>>>>>>>>> "2FB31EEC":
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
> >>>>>>>>>>>>>>>>>> 2FB31EEC
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in the
> >>>>>>>>>>>>>>>>>> last two
> >>>>>>>>>>>>>>>>>> lines.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> >>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> showing up?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers.
> >>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
we'd
> >>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across 2
> >>>>>>>>>>>>>>>>>>> files. Do we have existing
> >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
> >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> >>>>>>>>>>>>>>>>>>> temperature or something?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> >>>>>>>>>>>>>>>>>>>> sec).
> >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in the
> >>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content of
> >>>>>>>>>>>>>>>>>>>>> obs_hid
> >>>>>>>>>>>>>>>>>>>>> variable).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
> >>>>>>>>>>>>>>>>>>>>> (test
> >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
> >>>>>>>>>>>>>>>>>>>>> added one
> >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change the
> >>>>>>>>>>>>>>>>>>>>> nightly
> >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
> >>>>>>>>>>>>>>>>>>>>>> data.
> >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The output is available
> >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
> >>>>>>>>>>>>>>>>>>>>>> Gotway <
> >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic for
> >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr counts:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
> >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to dump
to
> >>>>>>>>>>>>>>>>>>>>>>> ascii
> >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000 string
> >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC not
> >>>>>>>>>>>>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
> >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John Halley
> >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard Soh
<
> >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
> >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> >>>>>>>>>>>>>>>>>>>>>>>>> from aqm.20180528/aqm.t12z.prepbufr.tm00.
> >>>>>>>>>>>>>>>>>>>>>>>>> There
> >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
I'm
> >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
take
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC. I
> >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >>>>>>>>>>>>>>>>>>>>>>>>>> \
> >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
> >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end = 0;
}
> >>>>>>>>>>>>>>>>>>>>>>>>>> //
> >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what you'd
> >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the "AIRNOW_MAX_240000"
> >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused by
> >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW PrepBUFR
> >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
hour
> >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is reasonable
for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
than
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295 currently)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for summary)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable. The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180622
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all of
the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
them
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table pb2nc_output.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Thu May 02 15:43:11 2019
Hi Perry -
Are you running on theia?
John suggests that we try to troubleshoot there ...
thanks,
David
On Thu May 02 15:35:10 2019, fillmore wrote:
> Hi Perry -
>
> An update on where we are on this issue:
> Using the prepda.2019042312.nc file you provided we indeed reproduce
> the point_stat error.
> The point_stat error is due to only a single entry the valid time
> table,
> e.g. from ncdump -v hdr_vld_table
> hdr_vld_table =
> "20190423_050000",
> while the hdr_vld indices in that file are hdr_vld = 0, 33, 33, ...
> which are all out of range except the first.
>
> If I run pb2nc here with your PREPBUFR data and your PB2NCConfig_aqm
> config file, however,
> I generate a different prepda.2019042312.nc file that runs through
to
> completion with point_stat.
> Its valid time table:
>
> hdr_vld_table =
> "20190423_030000",
> "20190423_010000",
> "20190423_020000",
> "20190423_230000",
> "20190423_150000",
> "20190423_090000",
> "20190423_070000",
> "20190423_060000",
> "20190423_210000",
> "20190423_050000",
> "20190423_170000",
> "20190423_100000",
> "20190423_200000",
> "20190423_140000",
> "20190423_130000",
> "20190423_160000",
> "20190423_040000",
> "20190423_180000",
> "20190423_080000",
> "20190423_120000",
> "20190423_110000",
> "20190423_220000",
> "20190423_190000",
> "20190424_000000" ;
> }
>
> So it seems pb2nc is running correctly here, but not on your side.
> Let me discuss with John the best way to proceed ...
> thanks,
> David
>
>
> On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > Hi, everyone,
> >
> > I am attaching this to this long thread regarding the daily max of
> > ozone
> > verification, so we remember what we were doing back then.
> >
> > I am now running this again using MET 8.0, and something seems to
not
> > be
> > working when I run point_stat. I get this error when running a
daily
> > max
> > of ozone:
> >
> > ERROR :
> > ERROR : StringArray::operator[](int) const -> range check error!
> > ERROR :
> >
> > I am using my own scripts to run this using MET, so this is not
using
> > METplus.
> >
> > You can find a log file of this here:
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > It
> > runs pb2nc and point_stat for max ozone. The pb2nc code is working
> > fine, I
> > believe.
> >
> > The script I am using to run this is here:
> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> >
> > The prepbufr files are:
> >
> > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> >
> > The model file is here:
> >
> >
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> >
> >
> > What could be causing this error?
> >
> > Thanks!
> >
> > Perry
> >
> >
> > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Great, glad that did the trick.
> > >
> > > John
> > >
> > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > > perry.shafran at noaa.gov> wrote:
> > >
> > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
> > >> avg.
> > >>
> > >> Thanks!
> > >>
> > >> Perry
> > >>
> > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > >> <johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >>> Perry,
> > >>>
> > >>> I think the problem is in your setting of
> > >>> valid_time="${CUR_VX_TIME}";
> > >>>
> > >>> Here's your wgrib2 output:
> > >>> wgrib2 oz_DAY1
> > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
> > >>> fcst:
> > >>>
> > >>> The "d=2018081112" tells you the model initialization time.
The
> > >>> valid
> > >>> time = init time + forecast time. So the OZMAX1 data is valid
16
> > >>> hours
> > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> > >>> wgrib2
> > >>> command using the "-vt" argument to list the verification
time.
> > >>>
> > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > >>> valid
> > >>> at the time you're requesting.
> > >>>
> > >>> John
> > >>>
> > >>>
> > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate
<
> > >>> perry.shafran at noaa.gov> wrote:
> > >>>
> > >>>> Hi, John,
> > >>>>
> > >>>> I haven't been able to work on this for awhile, but now I'm
> > >>>> working on
> > >>>> it again but seem to be running into issues.
> > >>>>
> > >>>> I have a file that has both the daily max for 1-hr ozone and
the
> > >>>> daily
> > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > >>>>
> > >>>> 9 (t10a2)
> > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>> >
> > >>>> wgrib2 oz_DAY1
> > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > >>>> 10 (t10a2)
> > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>> >
> > >>>>
> > >>>> As you can see they have different forecast times. But when
I
> > >>>> run
> > >>>> point_stat, I get this output:
> > >>>>
> > >>>> DEBUG 2:
> > >>>> DEBUG 2:
> > >>>>
--------------------------------------------------------------------------------
> > >>>> DEBUG 2:
> > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > >>>> WARNING:
> > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> > >>>> records
> > >>>> found for 'OZMAX8/L1'
> > >>>> WARNING:
> > >>>> WARNING:
> > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> > >>>> OZMAX8/L1
> > >>>> found in file: oz_DAY1
> > >>>> WARNING:
> > >>>>
> > >>>> Here is the relevant portion of the point_stat config file:
> > >>>>
> > >>>> fcst = {
> > >>>> field = [
> > >>>> { name = "OZMAX1"; level = "L1";
> > >>>> valid_time="${CUR_VX_TIME}"; },
> > >>>> { name = "OZMAX8"; level = "L1";
> > >>>> valid_time="${CUR_VX_TIME}"; }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> obs = {
> > >>>> convert(x) = x * 10^9;
> > >>>> field = [
> > >>>> { name = "COPO"; level = "A1"; message_type =
> > >>>> "AIRNOW_MAX_230000"; },
> > >>>> { name = "COPO"; level = "A8"; message_type =
> > >>>> "AIRNOW_MAX_230000"; }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> I don't remember if I had this working previously for both
the
> > >>>> 1hr
> > >>>> and
> > >>>> the 8hr ozone, but apparently there is some sort of issue for
> > >>>> the
> > >>>> 8hr
> > >>>> ozone. Have any thoughts?
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>> Perry
> > >>>>
> > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate
> > >>>> <
> > >>>> perry.shafran at noaa.gov> wrote:
> > >>>>
> > >>>>> I chose option 2, and that seems to work - I have a .stat
file
> > >>>>> with
> > >>>>> output. Now I'll need to do some work here to see if the
> > >>>>> output
> > >>>>> looks OK,
> > >>>>> and to configure the verification so it verifies OZMAX1 and
> > >>>>> OZMAX8.
> > >>>>>
> > >>>>> I also wonder if this would be easily modified for PM.
> > >>>>>
> > >>>>> Perry
> > >>>>>
> > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > >>>>> <johnhg at ucar.edu>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Perry,
> > >>>>>>
> > >>>>>> Yep, you've configured/run PB2NC to compute time summaries
> > >>>>>> that
> > >>>>>> are
> > >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> > >>>>>> purportedly
> > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > >>>>>> rejected due
> > >>>>>> to the valid time.
> > >>>>>>
> > >>>>>> It's up to you guys to decide what the "correct" way of
> > >>>>>> handling
> > >>>>>> this
> > >>>>>> is... but here are a couple of options:
> > >>>>>>
> > >>>>>> (1) You could use the existing setup and address this logic
in
> > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > >>>>>> from 0 to:
> > >>>>>>
> > >>>>>> obs_window = {
> > >>>>>> beg = 3600;
> > >>>>>> end = 3600;
> > >>>>>> }
> > >>>>>> This tells Point-Stat to match observations which are
exactly
> > >>>>>> 1-
> > >>>>>> hour
> > >>>>>> prior to the valid time of the forecast.
> > >>>>>>
> > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z
> > >>>>>> to
> > >>>>>> 05Z
> > >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> > >>>>>>
> > >>>>>> time_summary = {
> > >>>>>> flag = TRUE;
> > >>>>>> beg = "050000";
> > >>>>>> end = "050000";
> > >>>>>> ...
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> John
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > >>>>>> Affiliate
> > >>>>>> <
> > >>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>
> > >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
> > >>>>>>> it's
> > >>>>>>> empty. Here's the output from point_stat:
> > >>>>>>>
> > >>>>>>> DEBUG 4:
> > >>>>>>> DEBUG 4: Data plane information:
> > >>>>>>> DEBUG 4: plane min: 13.86
> > >>>>>>> DEBUG 4: plane max: 130.82
> > >>>>>>> DEBUG 4: scan mode: 64
> > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > >>>>>>> DEBUG 4: lead time: 170000
> > >>>>>>> DEBUG 4: init time: 20180708_120000
> > >>>>>>> DEBUG 4: bitmap flag: 255
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > >>>>>>> type
> > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > >>>>>>> NEAREST(1),
> > >>>>>>> using 0 pairs.
> > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > >>>>>>> DEBUG 3: Observations processed = 124637
> > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > >>>>>>>
> > >>>>>>> Looks like most rejections were due to valid time.
> > >>>>>>>
> > >>>>>>> Have any thoughts about this one?
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>>
> > >>>>>>> Perry
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> > >>>>>>> Affiliate
> > >>>>>>> <
> > >>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>
> > >>>>>>>> Hi, John,
> > >>>>>>>>
> > >>>>>>>> Looks like the changing of the time stamp from _04 to _05
> > >>>>>>>> did
> > >>>>>>>> the
> > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > >>>>>>>>
> > >>>>>>>> Thanks!
> > >>>>>>>>
> > >>>>>>>> Perry
> > >>>>>>>>
> > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > >>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>
> > >>>>>>>>> Perry,
> > >>>>>>>>>
> > >>>>>>>>> I agree, there's likely some issue with requesting the
> > >>>>>>>>> timestamp
> > >>>>>>>>> of the data from that file. I think switching the
> > >>>>>>>>> verification time to
> > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > >>>>>>>>>
> > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > >>>>>>>>> times
> > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > >>>>>>>>> and
> > >>>>>>>>> MET commands
> > >>>>>>>>> to illustrate.
> > >>>>>>>>>
> > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
> > >>>>>>>>> time
> > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> > >>>>>>>>> data file you
> > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> > >>>>>>>>> so
> > >>>>>>>>> I have that
> > >>>>>>>>> funky forecast hour:
> > >>>>>>>>>
> > >>>>>>>>> wgrib2 -t -ftime -vt
> > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > >>>>>>>>> fcst:vt=2018070505
> > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
> > >>>>>>>>>
> > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
> > >>>>>>>>> level
> > >>>>>>>>> (-v
> > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > >>>>>>>>> reading:
> > >>>>>>>>>
> > >>>>>>>>> # Read the first record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > >>>>>>>>> DEBUG 4: lead time: 230000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>> ...
> > >>>>>>>>> # Read the second record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > >>>>>>>>> DEBUG 4: lead time: 470000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>>
> > >>>>>>>>> Hope that helps.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> John
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> > >>>>>>>>> Affiliate <
> > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, John,
> > >>>>>>>>>>
> > >>>>>>>>>> When I implemented all the changes (and also set it up
so
> > >>>>>>>>>> it's
> > >>>>>>>>>> reading the beta6 version), I used this command:
> > >>>>>>>>>>
> > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > >>>>>>>>>>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > >>>>>>>>>> -v 4
> > >>>>>>>>>>
> > >>>>>>>>>> But then I got this error:
> > >>>>>>>>>>
> > >>>>>>>>>> DEBUG 2:
> > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> > >>>>>>>>>> matching
> > >>>>>>>>>> records found for 'OZMAX1/L1'
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> ERROR :
> > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
> > >>>>>>>>>> forecast
> > >>>>>>>>>> data
> > >>>>>>>>>> found! Exiting...
> > >>>>>>>>>>
> > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the
> > >>>>>>>>>> file
> > >>>>>>>>>> in this case reads:
> > >>>>>>>>>>
> > >>>>>>>>>> 14 (t14a2)
> > >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>
> > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > >>>>>>>>>> 15 (t14a2)
> > >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>>>>>>>>
> > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> > >>>>>>>>>> suggested
> > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> > >>>>>>>>>> it needs to be
> > >>>>>>>>>> the time stamp of the first day?
> > >>>>>>>>>>
> > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for any thoughts.
> > >>>>>>>>>>
> > >>>>>>>>>> Perry
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Perry,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm confused. In the previous air quality vx we did,
we
> > >>>>>>>>>>> used
> > >>>>>>>>>>> observation files named:
> > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > >>>>>>>>>>>
> > >>>>>>>>>>> And those files don't exist in the July data you just
> > >>>>>>>>>>> sent.
> > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm not sure if you're actually verifying against the
> > >>>>>>>>>>> BUFR
> > >>>>>>>>>>> or
> > >>>>>>>>>>> PREPBUFR.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > >>>>>>>>>>> settings...
> > >>>>>>>>>>>
> > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
> > >>>>>>>>>>> files
> > >>>>>>>>>>> -
> > >>>>>>>>>>> thecurrent day and the previous day...
> > >>>>>>>>>>> pb2nc \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > >>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>
> > >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
> > >>>>>>>>>>> like
> > >>>>>>>>>>> this:
> > >>>>>>>>>>> time_summary = {
> > >>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>> end = "040000";
> > >>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
> > >>>>>>>>>>> previous
> > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > >>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > >>>>>>>>>>> type = [ "max", "mean" ];
> > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at
> > >>>>>>>>>>> least
> > >>>>>>>>>>> 75% of expect obs are present
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > >>>>>>>>>>> over the
> > >>>>>>>>>>> defined time period. They're identified in the output
by
> > >>>>>>>>>>> the message type:
> > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > >>>>>>>>>>>
> > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > >>>>>>>>>>> sections like this for the max of the 1-hour data.
> > >>>>>>>>>>> Instead
> > >>>>>>>>>>> of specifying
> > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> > >>>>>>>>>>> to specify the
> > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > >>>>>>>>>>> file, I'd suggest
> > >>>>>>>>>>> using environment variables:
> > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > >>>>>>>>>>>
> > >>>>>>>>>>> fcst = {
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>> obs = {
> > >>>>>>>>>>> convert(x) = x * 10^9;
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Sounds good, thanks!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> John
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> > >>>>>>>>>>>> Affiliate <
> > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and
> > >>>>>>>>>>>>> focus on
> > >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
> > >>>>>>>>>>>>> over
> > >>>>>>>>>>>>> to Theia so we'll
> > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> > >>>>>>>>>>>>> into problems, I'll
> > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
> > >>>>>>>>>>>>> max/avg files).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for
> > >>>>>>>>>>>>>> bias
> > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>> may want to provide
> > >>>>>>>>>>>>>> him para5 runs.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> > >>>>>>>>>>>>>> to do
> > >>>>>>>>>>>>>> some testing some production files. Do you think I
> > >>>>>>>>>>>>>> should use parallels
> > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> > >>>>>>>>>>>>>> creating a MET+ use case
> > >>>>>>>>>>>>>> for AQ?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > >>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>> be OK for now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
> > >>>>>>>>>>>>>>> We
> > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > >>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>> management did not
> > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> > >>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > >>>>>>>>>>>>>>> parallel CMAQ data
> > >>>>>>>>>>>>>>> rather than production CMAQ.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I don't think that the MET team has any other
> > >>>>>>>>>>>>>>> versions
> > >>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can
> > >>>>>>>>>>>>>>> also clarify this as
> > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> current wgrib2?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > >>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
> > >>>>>>>>>>>>>>>> did
> > >>>>>>>>>>>>>>>> not want to risk any
> > >>>>>>>>>>>>>>>> further delays so we will have to do so with the
> > >>>>>>>>>>>>>>>> FYQ4
> > >>>>>>>>>>>>>>>> implementation.
> > >>>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
> > >>>>>>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> > >>>>>>>>>>>>>>>> but I don't know if I
> > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
> > >>>>>>>>>>>>>>>> sure.
> > >>>>>>>>>>>>>>>> But he would have
> > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
> > >>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>> this.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
> > >>>>>>>>>>>>>>>> talks
> > >>>>>>>>>>>>>>>> about?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Perry,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
> > >>>>>>>>>>>>>>>>> compute
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
> > >>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>> ozone concentrations.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Using the attached config files.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
> > >>>>>>>>>>>>>>>>> FULL,
> > >>>>>>>>>>>>>>>>> for interpolation
> > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> "max"
> > >>>>>>>>>>>>>>>>> file:
> > >>>>>>>>>>>>>>>>> wgrib2
> > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-
40
> > >>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>> ave fcst:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
> > >>>>>>>>>>>>>>>>> revealed
> > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17
> > >>>>>>>>>>>>>>>>> and 41, respectively.
> > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> > >>>>>>>>>>>>>>>>> file, we
> > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> use. Do this by
> > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > >>>>>>>>>>>>>>>>> fcst = {
> > >>>>>>>>>>>>>>>>> field = [
> > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time
> > >>>>>>>>>>>>>>>>> =
> > >>>>>>>>>>>>>>>>> "41";' }
> > >>>>>>>>>>>>>>>>> ];
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off
> > >>>>>>>>>>>>>>>>> by 1
> > >>>>>>>>>>>>>>>>> hour.
> > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
> > >>>>>>>>>>>>>>>>> time
> > >>>>>>>>>>>>>>>>> =
> > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> > >>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> > >>>>>>>>>>>>>>>>> file by
> > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > >>>>>>>>>>>>>>>>> obs_window = {
> > >>>>>>>>>>>>>>>>> beg = -3600;
> > >>>>>>>>>>>>>>>>> end = -3600;
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> So what're the next steps?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> > >>>>>>>>>>>>>>>>>> found
> > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
> > >>>>>>>>>>>>>>>>>> here:
> > >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-
> > >>>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> > >>>>>>>>>>>>>>>>>> to an ascii file using
> > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count
> > >>>>>>>>>>>>>>>>>> up
> > >>>>>>>>>>>>>>>>>> the number of regular
> > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
|
> > >>>>>>>>>>>>>>>>>> wc
> > >>>>>>>>>>>>>>>>>> -l
> > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
|
> > >>>>>>>>>>>>>>>>>> sort -u
> > >>>>>>>>>>>>>>>>>> | wc -l
> > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
wc
> > >>>>>>>>>>>>>>>>>> -l
> > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
> > >>>>>>>>>>>>>>>>>> sort
> > >>>>>>>>>>>>>>>>>> -u |
> > >>>>>>>>>>>>>>>>>> wc -l
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
> > >>>>>>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
> > >>>>>>>>>>>>>>>>>> meaning
> > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > >>>>>>>>>>>>>>>>>> duplicates.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is
> > >>>>>>>>>>>>>>>>>> named
> > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
> > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the
> > >>>>>>>>>>>>>>>>>> last two
> > >>>>>>>>>>>>>>>>>> lines.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> > >>>>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>> showing up?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Cheers.
> > >>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
> > >>>>>>>>>>>>>>>>>>> we'd
> > >>>>>>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across
2
> > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
> > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > >>>>>>>>>>>>>>>>>>> temperature or something?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > >>>>>>>>>>>>>>>>>>>> sec).
> > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the
> > >>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content
of
> > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > >>>>>>>>>>>>>>>>>>>>> variable).
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
> > >>>>>>>>>>>>>>>>>>>>> (test
> > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
> > >>>>>>>>>>>>>>>>>>>>> added one
> > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> > >>>>>>>>>>>>>>>>>>>>> nightly
> > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
> > >>>>>>>>>>>>>>>>>>>>>> data.
> > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
> > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
> > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > >>>>>>>>>>>>>>>>>>>>>>> actually
> > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
> > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
> > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
> > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
> > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC.
I
> > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > >>>>>>>>>>>>>>>>>>>>>>>>>> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
> > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end =
0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
> > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota
in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180622
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri May 03 06:35:41 2019
Hi, David, John,
I am not running on Theia. In fact I think it would take quite a bit
of
effort to transition this thing to Theia, and I'll need to be running
this
verification on prod data anyway, which is only on WCOSS.
However, if you wish to troubleshoot on Theia, you should do so, and I
think you have all you need to be able to do so, right?
It would seem that the error on my side is in PB2NC, so I'll probably
need
to make sure I'm running what you're running.
Thanks!
Perry
On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT
<met_help at ucar.edu>
wrote:
> Hi Perry -
> Are you running on theia?
> John suggests that we try to troubleshoot there ...
> thanks,
> David
>
> On Thu May 02 15:35:10 2019, fillmore wrote:
> > Hi Perry -
> >
> > An update on where we are on this issue:
> > Using the prepda.2019042312.nc file you provided we indeed
reproduce
> > the point_stat error.
> > The point_stat error is due to only a single entry the valid time
> > table,
> > e.g. from ncdump -v hdr_vld_table
> > hdr_vld_table =
> > "20190423_050000",
> > while the hdr_vld indices in that file are hdr_vld = 0, 33, 33,
...
> > which are all out of range except the first.
> >
> > If I run pb2nc here with your PREPBUFR data and your
PB2NCConfig_aqm
> > config file, however,
> > I generate a different prepda.2019042312.nc file that runs through
to
> > completion with point_stat.
> > Its valid time table:
> >
> > hdr_vld_table =
> > "20190423_030000",
> > "20190423_010000",
> > "20190423_020000",
> > "20190423_230000",
> > "20190423_150000",
> > "20190423_090000",
> > "20190423_070000",
> > "20190423_060000",
> > "20190423_210000",
> > "20190423_050000",
> > "20190423_170000",
> > "20190423_100000",
> > "20190423_200000",
> > "20190423_140000",
> > "20190423_130000",
> > "20190423_160000",
> > "20190423_040000",
> > "20190423_180000",
> > "20190423_080000",
> > "20190423_120000",
> > "20190423_110000",
> > "20190423_220000",
> > "20190423_190000",
> > "20190424_000000" ;
> > }
> >
> > So it seems pb2nc is running correctly here, but not on your side.
> > Let me discuss with John the best way to proceed ...
> > thanks,
> > David
> >
> >
> > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > Hi, everyone,
> > >
> > > I am attaching this to this long thread regarding the daily max
of
> > > ozone
> > > verification, so we remember what we were doing back then.
> > >
> > > I am now running this again using MET 8.0, and something seems
to not
> > > be
> > > working when I run point_stat. I get this error when running a
daily
> > > max
> > > of ozone:
> > >
> > > ERROR :
> > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > ERROR :
> > >
> > > I am using my own scripts to run this using MET, so this is not
using
> > > METplus.
> > >
> > > You can find a log file of this here:
> > >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > It
> > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> > > fine, I
> > > believe.
> > >
> > > The script I am using to run this is here:
> > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > >
> > > The prepbufr files are:
> > >
> > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > >
> > > The model file is here:
> > >
> > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > >
> > >
> > > What could be causing this error?
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > >
> > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Great, glad that did the trick.
> > > >
> > > > John
> > > >
> > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate
<
> > > > perry.shafran at noaa.gov> wrote:
> > > >
> > > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-
hr
> > > >> avg.
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Perry
> > > >>
> > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > >> <johnhg at ucar.edu>
> > > >> wrote:
> > > >>
> > > >>> Perry,
> > > >>>
> > > >>> I think the problem is in your setting of
> > > >>> valid_time="${CUR_VX_TIME}";
> > > >>>
> > > >>> Here's your wgrib2 output:
> > > >>> wgrib2 oz_DAY1
> > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
> > > >>> fcst:
> > > >>>
> > > >>> The "d=2018081112" tells you the model initialization time.
The
> > > >>> valid
> > > >>> time = init time + forecast time. So the OZMAX1 data is
valid 16
> > > >>> hours
> > > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> > > >>> wgrib2
> > > >>> command using the "-vt" argument to list the verification
time.
> > > >>>
> > > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > > >>> valid
> > > >>> at the time you're requesting.
> > > >>>
> > > >>> John
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate <
> > > >>> perry.shafran at noaa.gov> wrote:
> > > >>>
> > > >>>> Hi, John,
> > > >>>>
> > > >>>> I haven't been able to work on this for awhile, but now I'm
> > > >>>> working on
> > > >>>> it again but seem to be running into issues.
> > > >>>>
> > > >>>> I have a file that has both the daily max for 1-hr ozone
and the
> > > >>>> daily
> > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > >>>>
> > > >>>> 9 (t10a2)
> > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>> >
> > > >>>> wgrib2 oz_DAY1
> > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > >>>> 10 (t10a2)
> > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>> >
> > > >>>>
> > > >>>> As you can see they have different forecast times. But
when I
> > > >>>> run
> > > >>>> point_stat, I get this output:
> > > >>>>
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2:
> > > >>>>
>
--------------------------------------------------------------------------------
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > >>>> WARNING:
> > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> > > >>>> records
> > > >>>> found for 'OZMAX8/L1'
> > > >>>> WARNING:
> > > >>>> WARNING:
> > > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> > > >>>> OZMAX8/L1
> > > >>>> found in file: oz_DAY1
> > > >>>> WARNING:
> > > >>>>
> > > >>>> Here is the relevant portion of the point_stat config file:
> > > >>>>
> > > >>>> fcst = {
> > > >>>> field = [
> > > >>>> { name = "OZMAX1"; level = "L1";
> > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > >>>> { name = "OZMAX8"; level = "L1";
> > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> obs = {
> > > >>>> convert(x) = x * 10^9;
> > > >>>> field = [
> > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; },
> > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> I don't remember if I had this working previously for both
the
> > > >>>> 1hr
> > > >>>> and
> > > >>>> the 8hr ozone, but apparently there is some sort of issue
for
> > > >>>> the
> > > >>>> 8hr
> > > >>>> ozone. Have any thoughts?
> > > >>>>
> > > >>>> Thanks!
> > > >>>>
> > > >>>> Perry
> > > >>>>
> > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate
> > > >>>> <
> > > >>>> perry.shafran at noaa.gov> wrote:
> > > >>>>
> > > >>>>> I chose option 2, and that seems to work - I have a .stat
file
> > > >>>>> with
> > > >>>>> output. Now I'll need to do some work here to see if the
> > > >>>>> output
> > > >>>>> looks OK,
> > > >>>>> and to configure the verification so it verifies OZMAX1
and
> > > >>>>> OZMAX8.
> > > >>>>>
> > > >>>>> I also wonder if this would be easily modified for PM.
> > > >>>>>
> > > >>>>> Perry
> > > >>>>>
> > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > >>>>> <johnhg at ucar.edu>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Perry,
> > > >>>>>>
> > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > >>>>>> that
> > > >>>>>> are
> > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that are
> > > >>>>>> purportedly
> > > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > > >>>>>> rejected due
> > > >>>>>> to the valid time.
> > > >>>>>>
> > > >>>>>> It's up to you guys to decide what the "correct" way of
> > > >>>>>> handling
> > > >>>>>> this
> > > >>>>>> is... but here are a couple of options:
> > > >>>>>>
> > > >>>>>> (1) You could use the existing setup and address this
logic in
> > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > > >>>>>> from 0 to:
> > > >>>>>>
> > > >>>>>> obs_window = {
> > > >>>>>> beg = 3600;
> > > >>>>>> end = 3600;
> > > >>>>>> }
> > > >>>>>> This tells Point-Stat to match observations which are
exactly
> > > >>>>>> 1-
> > > >>>>>> hour
> > > >>>>>> prior to the valid time of the forecast.
> > > >>>>>>
> > > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z
> > > >>>>>> to
> > > >>>>>> 05Z
> > > >>>>>> instead and then leave the Point-Stat obs_window setting
at 0:
> > > >>>>>>
> > > >>>>>> time_summary = {
> > > >>>>>> flag = TRUE;
> > > >>>>>> beg = "050000";
> > > >>>>>> end = "050000";
> > > >>>>>> ...
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> John
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > > >>>>>> Affiliate
> > > >>>>>> <
> > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>
> > > >>>>>>> Oops, spoke too soon. There *is* a .stat output file,
but
> > > >>>>>>> it's
> > > >>>>>>> empty. Here's the output from point_stat:
> > > >>>>>>>
> > > >>>>>>> DEBUG 4:
> > > >>>>>>> DEBUG 4: Data plane information:
> > > >>>>>>> DEBUG 4: plane min: 13.86
> > > >>>>>>> DEBUG 4: plane max: 130.82
> > > >>>>>>> DEBUG 4: scan mode: 64
> > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > >>>>>>> DEBUG 4: lead time: 170000
> > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > > >>>>>>> type
> > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > > >>>>>>> NEAREST(1),
> > > >>>>>>> using 0 pairs.
> > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > >>>>>>>
> > > >>>>>>> Looks like most rejections were due to valid time.
> > > >>>>>>>
> > > >>>>>>> Have any thoughts about this one?
> > > >>>>>>>
> > > >>>>>>> Thanks!
> > > >>>>>>>
> > > >>>>>>> Perry
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> > > >>>>>>> Affiliate
> > > >>>>>>> <
> > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi, John,
> > > >>>>>>>>
> > > >>>>>>>> Looks like the changing of the time stamp from _04 to
_05
> > > >>>>>>>> did
> > > >>>>>>>> the
> > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > >>>>>>>>
> > > >>>>>>>> Thanks!
> > > >>>>>>>>
> > > >>>>>>>> Perry
> > > >>>>>>>>
> > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Perry,
> > > >>>>>>>>>
> > > >>>>>>>>> I agree, there's likely some issue with requesting the
> > > >>>>>>>>> timestamp
> > > >>>>>>>>> of the data from that file. I think switching the
> > > >>>>>>>>> verification time to
> > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > >>>>>>>>>
> > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > > >>>>>>>>> times
> > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > > >>>>>>>>> and
> > > >>>>>>>>> MET commands
> > > >>>>>>>>> to illustrate.
> > > >>>>>>>>>
> > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > >>>>>>>>> time
> > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> > > >>>>>>>>> data file you
> > > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> > > >>>>>>>>> so
> > > >>>>>>>>> I have that
> > > >>>>>>>>> funky forecast hour:
> > > >>>>>>>>>
> > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > >>>>>>>>> fcst:vt=2018070505
> > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > >>>>>>>>>
> > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
> > > >>>>>>>>> level
> > > >>>>>>>>> (-v
> > > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > > >>>>>>>>> reading:
> > > >>>>>>>>>
> > > >>>>>>>>> # Read the first record
> > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>> ...
> > > >>>>>>>>> # Read the second record
> > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>>
> > > >>>>>>>>> Hope that helps.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> John
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> > > >>>>>>>>> Affiliate <
> > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi, John,
> > > >>>>>>>>>>
> > > >>>>>>>>>> When I implemented all the changes (and also set it
up so
> > > >>>>>>>>>> it's
> > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > >>>>>>>>>>
> > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > >>>>>>>>>>
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > >>>>>>>>>> -v 4
> > > >>>>>>>>>>
> > > >>>>>>>>>> But then I got this error:
> > > >>>>>>>>>>
> > > >>>>>>>>>> DEBUG 2:
> > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> > > >>>>>>>>>> matching
> > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> ERROR :
> > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
> > > >>>>>>>>>> forecast
> > > >>>>>>>>>> data
> > > >>>>>>>>>> found! Exiting...
> > > >>>>>>>>>>
> > > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output
for the
> > > >>>>>>>>>> file
> > > >>>>>>>>>> in this case reads:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 14 (t14a2)
> > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>>>>>>>>> 15 (t14a2)
> > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>>>>>>>>
> > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> > > >>>>>>>>>> suggested
> > > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> > > >>>>>>>>>> it needs to be
> > > >>>>>>>>>> the time stamp of the first day?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for any thoughts.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Perry
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Perry,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did, we
> > > >>>>>>>>>>> used
> > > >>>>>>>>>>> observation files named:
> > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> And those files don't exist in the July data you
just
> > > >>>>>>>>>>> sent.
> > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm not sure if you're actually verifying against
the
> > > >>>>>>>>>>> BUFR
> > > >>>>>>>>>>> or
> > > >>>>>>>>>>> PREPBUFR.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > > >>>>>>>>>>> settings...
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
> > > >>>>>>>>>>> files
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>> thecurrent day and the previous day...
> > > >>>>>>>>>>> pb2nc \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > >>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > >>>>>>>>>>> like
> > > >>>>>>>>>>> this:
> > > >>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
> > > >>>>>>>>>>> previous
> > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > >>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries
when at
> > > >>>>>>>>>>> least
> > > >>>>>>>>>>> 75% of expect obs are present
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > > >>>>>>>>>>> over the
> > > >>>>>>>>>>> defined time period. They're identified in the
output by
> > > >>>>>>>>>>> the message type:
> > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > > >>>>>>>>>>> sections like this for the max of the 1-hour data.
> > > >>>>>>>>>>> Instead
> > > >>>>>>>>>>> of specifying
> > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> > > >>>>>>>>>>> to specify the
> > > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > > >>>>>>>>>>> file, I'd suggest
> > > >>>>>>>>>>> using environment variables:
> > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> fcst = {
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>> obs = {
> > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Sounds good, thanks!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> John
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > >>>>>>>>>>>> Affiliate <
> > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and
> > > >>>>>>>>>>>>> focus on
> > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files
> > > >>>>>>>>>>>>> over
> > > >>>>>>>>>>>>> to Theia so we'll
> > > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> > > >>>>>>>>>>>>> into problems, I'll
> > > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-
hr
> > > >>>>>>>>>>>>> max/avg files).
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for
> > > >>>>>>>>>>>>>> bias
> > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe,
> > > >>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>> may want to provide
> > > >>>>>>>>>>>>>> him para5 runs.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> > > >>>>>>>>>>>>>> to do
> > > >>>>>>>>>>>>>> some testing some production files. Do you think
I
> > > >>>>>>>>>>>>>> should use parallels
> > > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> > > >>>>>>>>>>>>>> creating a MET+ use case
> > > >>>>>>>>>>>>>> for AQ?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > > >>>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>> be OK for now.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels.
> > > >>>>>>>>>>>>>>> We
> > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > > >>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>> management did not
> > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > >>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I don't think that the MET team has any other
> > > >>>>>>>>>>>>>>> versions
> > > >>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can
> > > >>>>>>>>>>>>>>> also clarify this as
> > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> current wgrib2?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > > >>>>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management
> > > >>>>>>>>>>>>>>>> did
> > > >>>>>>>>>>>>>>>> not want to risk any
> > > >>>>>>>>>>>>>>>> further delays so we will have to do so with
the
> > > >>>>>>>>>>>>>>>> FYQ4
> > > >>>>>>>>>>>>>>>> implementation.
> > > >>>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
> > > >>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> > > >>>>>>>>>>>>>>>> but I don't know if I
> > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
> > > >>>>>>>>>>>>>>>> sure.
> > > >>>>>>>>>>>>>>>> But he would have
> > > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
> > > >>>>>>>>>>>>>>>> approach
> > > >>>>>>>>>>>>>>>> this.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues
John
> > > >>>>>>>>>>>>>>>> talks
> > > >>>>>>>>>>>>>>>> about?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
> > > >>>>>>>>>>>>>>>>> compute
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and
8-
> > > >>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > >>>>>>>>>>>>>>>>> FULL,
> > > >>>>>>>>>>>>>>>>> for interpolation
> > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents
of
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>> "max"
> > > >>>>>>>>>>>>>>>>> file:
> > > >>>>>>>>>>>>>>>>> wgrib2
> > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
level:17-40
> > > >>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>> ave fcst:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > > >>>>>>>>>>>>>>>>> revealed
> > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours 17
> > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> > > >>>>>>>>>>>>>>>>> file, we
> > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> use. Do this by
> > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > >>>>>>>>>>>>>>>>> fcst = {
> > > >>>>>>>>>>>>>>>>> field = [
> > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time
> > > >>>>>>>>>>>>>>>>> =
> > > >>>>>>>>>>>>>>>>> "41";' }
> > > >>>>>>>>>>>>>>>>> ];
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps
are off
> > > >>>>>>>>>>>>>>>>> by 1
> > > >>>>>>>>>>>>>>>>> hour.
> > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead
> > > >>>>>>>>>>>>>>>>> time
> > > >>>>>>>>>>>>>>>>> =
> > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> > > >>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> > > >>>>>>>>>>>>>>>>> file by
> > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > > >>>>>>>>>>>>>>>>> obs_window = {
> > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > >>>>>>>>>>>>>>>>> end = -3600;
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> > > >>>>>>>>>>>>>>>>>> found
> > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
> > > >>>>>>>>>>>>>>>>>> here:
> > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-
> > > >>>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count
> > > >>>>>>>>>>>>>>>>>> up
> > > >>>>>>>>>>>>>>>>>> the number of regular
> > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > >>>>>>>>>>>>>>>>>> wc
> > > >>>>>>>>>>>>>>>>>> -l
> > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > >>>>>>>>>>>>>>>>>> sort -u
> > > >>>>>>>>>>>>>>>>>> | wc -l
> > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt
| wc
> > > >>>>>>>>>>>>>>>>>> -l
> > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt
|
> > > >>>>>>>>>>>>>>>>>> sort
> > > >>>>>>>>>>>>>>>>>> -u |
> > > >>>>>>>>>>>>>>>>>> wc -l
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning
that
> > > >>>>>>>>>>>>>>>>>> all
> > > >>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > > >>>>>>>>>>>>>>>>>> meaning
> > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > > >>>>>>>>>>>>>>>>>> duplicates.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is
> > > >>>>>>>>>>>>>>>>>> named
> > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the
> > > >>>>>>>>>>>>>>>>>> last two
> > > >>>>>>>>>>>>>>>>>> lines.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> > > >>>>>>>>>>>>>>>>>> are
> > > >>>>>>>>>>>>>>>>>> showing up?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > >>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason
why
> > > >>>>>>>>>>>>>>>>>>> we'd
> > > >>>>>>>>>>>>>>>>>>> have
> > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > > >>>>>>>>>>>>>>>>>>>> sec).
> > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the
> > > >>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content of
> > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> > > >>>>>>>>>>>>>>>>>>>>> (test
> > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs.
I
> > > >>>>>>>>>>>>>>>>>>>>> added one
> > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh
<
> > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute
24-hour
> > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
> > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump
> > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in
the
> > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00
and
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC. I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end
= 0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of
them to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a
24
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh
<
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0
to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180622
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some
of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced
by
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
>
>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Fri May 03 10:02:27 2019
Perry,
We can log on to wcoss and test there instead. Yes, pb2nc does seem
to be
the culprit. Can you point us the wcoss node you're using and the
working
directory there? We basically need to recreate your call to pb2nc to
try
to replicate the behavior.
Thanks,
John
On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, David, John,
>
> I am not running on Theia. In fact I think it would take quite a
bit of
> effort to transition this thing to Theia, and I'll need to be
running this
> verification on prod data anyway, which is only on WCOSS.
>
> However, if you wish to troubleshoot on Theia, you should do so, and
I
> think you have all you need to be able to do so, right?
>
> It would seem that the error on my side is in PB2NC, so I'll
probably need
> to make sure I'm running what you're running.
>
> Thanks!
>
> Perry
>
> On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Perry -
> > Are you running on theia?
> > John suggests that we try to troubleshoot there ...
> > thanks,
> > David
> >
> > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > Hi Perry -
> > >
> > > An update on where we are on this issue:
> > > Using the prepda.2019042312.nc file you provided we indeed
reproduce
> > > the point_stat error.
> > > The point_stat error is due to only a single entry the valid
time
> > > table,
> > > e.g. from ncdump -v hdr_vld_table
> > > hdr_vld_table =
> > > "20190423_050000",
> > > while the hdr_vld indices in that file are hdr_vld = 0, 33, 33,
...
> > > which are all out of range except the first.
> > >
> > > If I run pb2nc here with your PREPBUFR data and your
PB2NCConfig_aqm
> > > config file, however,
> > > I generate a different prepda.2019042312.nc file that runs
through to
> > > completion with point_stat.
> > > Its valid time table:
> > >
> > > hdr_vld_table =
> > > "20190423_030000",
> > > "20190423_010000",
> > > "20190423_020000",
> > > "20190423_230000",
> > > "20190423_150000",
> > > "20190423_090000",
> > > "20190423_070000",
> > > "20190423_060000",
> > > "20190423_210000",
> > > "20190423_050000",
> > > "20190423_170000",
> > > "20190423_100000",
> > > "20190423_200000",
> > > "20190423_140000",
> > > "20190423_130000",
> > > "20190423_160000",
> > > "20190423_040000",
> > > "20190423_180000",
> > > "20190423_080000",
> > > "20190423_120000",
> > > "20190423_110000",
> > > "20190423_220000",
> > > "20190423_190000",
> > > "20190424_000000" ;
> > > }
> > >
> > > So it seems pb2nc is running correctly here, but not on your
side.
> > > Let me discuss with John the best way to proceed ...
> > > thanks,
> > > David
> > >
> > >
> > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > Hi, everyone,
> > > >
> > > > I am attaching this to this long thread regarding the daily
max of
> > > > ozone
> > > > verification, so we remember what we were doing back then.
> > > >
> > > > I am now running this again using MET 8.0, and something seems
to not
> > > > be
> > > > working when I run point_stat. I get this error when running
a daily
> > > > max
> > > > of ozone:
> > > >
> > > > ERROR :
> > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > ERROR :
> > > >
> > > > I am using my own scripts to run this using MET, so this is
not using
> > > > METplus.
> > > >
> > > > You can find a log file of this here:
> > > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > It
> > > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> > > > fine, I
> > > > believe.
> > > >
> > > > The script I am using to run this is here:
> > > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > >
> > > > The prepbufr files are:
> > > >
> > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > >
> > > > The model file is here:
> > > >
> > > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > >
> > > >
> > > > What could be causing this error?
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > >
> > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > > > wrote:
> > > >
> > > > > Great, glad that did the trick.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
Affiliate <
> > > > > perry.shafran at noaa.gov> wrote:
> > > > >
> > > > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the
1-hr
> > > > >> avg.
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> Perry
> > > > >>
> > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > >> <johnhg at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >>> Perry,
> > > > >>>
> > > > >>> I think the problem is in your setting of
> > > > >>> valid_time="${CUR_VX_TIME}";
> > > > >>>
> > > > >>> Here's your wgrib2 output:
> > > > >>> wgrib2 oz_DAY1
> > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> > > > >>> fcst:
> > > > >>>
> > > > >>> The "d=2018081112" tells you the model initialization
time. The
> > > > >>> valid
> > > > >>> time = init time + forecast time. So the OZMAX1 data is
valid 16
> > > > >>> hours
> > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun that
> > > > >>> wgrib2
> > > > >>> command using the "-vt" argument to list the verification
time.
> > > > >>>
> > > > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > > > >>> valid
> > > > >>> at the time you're requesting.
> > > > >>>
> > > > >>> John
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate <
> > > > >>> perry.shafran at noaa.gov> wrote:
> > > > >>>
> > > > >>>> Hi, John,
> > > > >>>>
> > > > >>>> I haven't been able to work on this for awhile, but now
I'm
> > > > >>>> working on
> > > > >>>> it again but seem to be running into issues.
> > > > >>>>
> > > > >>>> I have a file that has both the daily max for 1-hr ozone
and the
> > > > >>>> daily
> > > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > > >>>>
> > > > >>>> 9 (t10a2)
> > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >>>> >
> > > > >>>> wgrib2 oz_DAY1
> > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > > >>>> 10 (t10a2)
> > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >>>> >
> > > > >>>>
> > > > >>>> As you can see they have different forecast times. But
when I
> > > > >>>> run
> > > > >>>> point_stat, I get this output:
> > > > >>>>
> > > > >>>> DEBUG 2:
> > > > >>>> DEBUG 2:
> > > > >>>>
> >
>
--------------------------------------------------------------------------------
> > > > >>>> DEBUG 2:
> > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > >>>> WARNING:
> > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > > >>>> records
> > > > >>>> found for 'OZMAX8/L1'
> > > > >>>> WARNING:
> > > > >>>> WARNING:
> > > > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> > > > >>>> OZMAX8/L1
> > > > >>>> found in file: oz_DAY1
> > > > >>>> WARNING:
> > > > >>>>
> > > > >>>> Here is the relevant portion of the point_stat config
file:
> > > > >>>>
> > > > >>>> fcst = {
> > > > >>>> field = [
> > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > >>>> ];
> > > > >>>> }
> > > > >>>>
> > > > >>>> obs = {
> > > > >>>> convert(x) = x * 10^9;
> > > > >>>> field = [
> > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > >>>> "AIRNOW_MAX_230000"; },
> > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > >>>> "AIRNOW_MAX_230000"; }
> > > > >>>> ];
> > > > >>>> }
> > > > >>>>
> > > > >>>> I don't remember if I had this working previously for
both the
> > > > >>>> 1hr
> > > > >>>> and
> > > > >>>> the 8hr ozone, but apparently there is some sort of issue
for
> > > > >>>> the
> > > > >>>> 8hr
> > > > >>>> ozone. Have any thoughts?
> > > > >>>>
> > > > >>>> Thanks!
> > > > >>>>
> > > > >>>> Perry
> > > > >>>>
> > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate
> > > > >>>> <
> > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>
> > > > >>>>> I chose option 2, and that seems to work - I have a
.stat file
> > > > >>>>> with
> > > > >>>>> output. Now I'll need to do some work here to see if
the
> > > > >>>>> output
> > > > >>>>> looks OK,
> > > > >>>>> and to configure the verification so it verifies OZMAX1
and
> > > > >>>>> OZMAX8.
> > > > >>>>>
> > > > >>>>> I also wonder if this would be easily modified for PM.
> > > > >>>>>
> > > > >>>>> Perry
> > > > >>>>>
> > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > > >>>>> <johnhg at ucar.edu>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Perry,
> > > > >>>>>>
> > > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > > >>>>>> that
> > > > >>>>>> are
> > > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that are
> > > > >>>>>> purportedly
> > > > >>>>>> valid at 05Z. And that's the reason why you have so
many obs
> > > > >>>>>> rejected due
> > > > >>>>>> to the valid time.
> > > > >>>>>>
> > > > >>>>>> It's up to you guys to decide what the "correct" way of
> > > > >>>>>> handling
> > > > >>>>>> this
> > > > >>>>>> is... but here are a couple of options:
> > > > >>>>>>
> > > > >>>>>> (1) You could use the existing setup and address this
logic in
> > > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > > > >>>>>> from 0 to:
> > > > >>>>>>
> > > > >>>>>> obs_window = {
> > > > >>>>>> beg = 3600;
> > > > >>>>>> end = 3600;
> > > > >>>>>> }
> > > > >>>>>> This tells Point-Stat to match observations which are
exactly
> > > > >>>>>> 1-
> > > > >>>>>> hour
> > > > >>>>>> prior to the valid time of the forecast.
> > > > >>>>>>
> > > > >>>>>> (2) You could rerun PB2NC to compute a time summary
from 05Z
> > > > >>>>>> to
> > > > >>>>>> 05Z
> > > > >>>>>> instead and then leave the Point-Stat obs_window
setting at 0:
> > > > >>>>>>
> > > > >>>>>> time_summary = {
> > > > >>>>>> flag = TRUE;
> > > > >>>>>> beg = "050000";
> > > > >>>>>> end = "050000";
> > > > >>>>>> ...
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>> John
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > > > >>>>>> Affiliate
> > > > >>>>>> <
> > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>
> > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output file,
but
> > > > >>>>>>> it's
> > > > >>>>>>> empty. Here's the output from point_stat:
> > > > >>>>>>>
> > > > >>>>>>> DEBUG 4:
> > > > >>>>>>> DEBUG 4: Data plane information:
> > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > > > >>>>>>> type
> > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > > > >>>>>>> NEAREST(1),
> > > > >>>>>>> using 0 pairs.
> > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > >>>>>>>
> > > > >>>>>>> Looks like most rejections were due to valid time.
> > > > >>>>>>>
> > > > >>>>>>> Have any thoughts about this one?
> > > > >>>>>>>
> > > > >>>>>>> Thanks!
> > > > >>>>>>>
> > > > >>>>>>> Perry
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> > > > >>>>>>> Affiliate
> > > > >>>>>>> <
> > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Hi, John,
> > > > >>>>>>>>
> > > > >>>>>>>> Looks like the changing of the time stamp from _04 to
_05
> > > > >>>>>>>> did
> > > > >>>>>>>> the
> > > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks!
> > > > >>>>>>>>
> > > > >>>>>>>> Perry
> > > > >>>>>>>>
> > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway
<
> > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Perry,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I agree, there's likely some issue with requesting
the
> > > > >>>>>>>>> timestamp
> > > > >>>>>>>>> of the data from that file. I think switching the
> > > > >>>>>>>>> verification time to
> > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > > > >>>>>>>>> times
> > > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > > > >>>>>>>>> and
> > > > >>>>>>>>> MET commands
> > > > >>>>>>>>> to illustrate.
> > > > >>>>>>>>>
> > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > > >>>>>>>>> time
> > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> > > > >>>>>>>>> data file you
> > > > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> > > > >>>>>>>>> so
> > > > >>>>>>>>> I have that
> > > > >>>>>>>>> funky forecast hour:
> > > > >>>>>>>>>
> > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > > >>>>>>>>> fcst:vt=2018070505
> > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > > >>>>>>>>>
> > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
verbosity
> > > > >>>>>>>>> level
> > > > >>>>>>>>> (-v
> > > > >>>>>>>>> 4) to have MET tell you the timing info of the data
it is
> > > > >>>>>>>>> reading:
> > > > >>>>>>>>>
> > > > >>>>>>>>> # Read the first record
> > > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > >>>>>>>>> ...
> > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > >>>>>>>>> ...
> > > > >>>>>>>>> # Read the second record
> > > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > >>>>>>>>> ...
> > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > >>>>>>>>>
> > > > >>>>>>>>> Hope that helps.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks,
> > > > >>>>>>>>> John
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> > > > >>>>>>>>> Affiliate <
> > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi, John,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> When I implemented all the changes (and also set it
up so
> > > > >>>>>>>>>> it's
> > > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > >>>>>>>>>>
> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > >>>>>>>>>> -v 4
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> But then I got this error:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> DEBUG 2:
> > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> > > > >>>>>>>>>> matching
> > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> ERROR :
> > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
> > > > >>>>>>>>>> forecast
> > > > >>>>>>>>>> data
> > > > >>>>>>>>>> found! Exiting...
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output
for the
> > > > >>>>>>>>>> file
> > > > >>>>>>>>>> in this case reads:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 14 (t14a2)
> > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > >>>>>>>>>> 15 (t14a2)
> > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> > > > >>>>>>>>>> suggested
> > > > >>>>>>>>>> that it should be the time stamp of the 2nd day,
but maybe
> > > > >>>>>>>>>> it needs to be
> > > > >>>>>>>>>> the time stamp of the first day?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks for any thoughts.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Perry
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway
<
> > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Perry,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did, we
> > > > >>>>>>>>>>> used
> > > > >>>>>>>>>>> observation files named:
> > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> And those files don't exist in the July data you
just
> > > > >>>>>>>>>>> sent.
> > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
aqm.t12z.anowpm.pb.tm00
> > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I'm not sure if you're actually verifying against
the
> > > > >>>>>>>>>>> BUFR
> > > > >>>>>>>>>>> or
> > > > >>>>>>>>>>> PREPBUFR.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > > > >>>>>>>>>>> settings...
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2
input
> > > > >>>>>>>>>>> files
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > >>>>>>>>>>> pb2nc \
> > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > >>>>>>>>>>> PB2NCConfig \
> > > > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > > >>>>>>>>>>> like
> > > > >>>>>>>>>>> this:
> > > > >>>>>>>>>>> time_summary = {
> > > > >>>>>>>>>>> flag = TRUE;
> > > > >>>>>>>>>>> beg = "040000";
> > > > >>>>>>>>>>> end = "040000";
> > > > >>>>>>>>>>> step = 24*3600;
> > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
> > > > >>>>>>>>>>> previous
> > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > >>>>>>>>>>> grib_code = [];
> > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every
hour
> > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries
when at
> > > > >>>>>>>>>>> least
> > > > >>>>>>>>>>> 75% of expect obs are present
> > > > >>>>>>>>>>> }
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > > > >>>>>>>>>>> over the
> > > > >>>>>>>>>>> defined time period. They're identified in the
output by
> > > > >>>>>>>>>>> the message type:
> > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst"
and "obs"
> > > > >>>>>>>>>>> sections like this for the max of the 1-hour data.
> > > > >>>>>>>>>>> Instead
> > > > >>>>>>>>>>> of specifying
> > > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> > > > >>>>>>>>>>> to specify the
> > > > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > > > >>>>>>>>>>> file, I'd suggest
> > > > >>>>>>>>>>> using environment variables:
> > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> fcst = {
> > > > >>>>>>>>>>> field = [
> > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > >>>>>>>>>>> ];
> > > > >>>>>>>>>>> }
> > > > >>>>>>>>>>> obs = {
> > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > >>>>>>>>>>> field = [
> > > > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type
=
> > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > >>>>>>>>>>> ];
> > > > >>>>>>>>>>> }
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway
<
> > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> John
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > > >>>>>>>>>>>> Affiliate <
> > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc
grids and
> > > > >>>>>>>>>>>>> focus on
> > > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files
> > > > >>>>>>>>>>>>> over
> > > > >>>>>>>>>>>>> to Theia so we'll
> > > > >>>>>>>>>>>>> let him work on the files he has for now and if
he runs
> > > > >>>>>>>>>>>>> into problems, I'll
> > > > >>>>>>>>>>>>> provide him with the para5 runs (at least the
24-hr
> > > > >>>>>>>>>>>>> max/avg files).
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not for
> > > > >>>>>>>>>>>>>> bias
> > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe,
> > > > >>>>>>>>>>>>>> you
> > > > >>>>>>>>>>>>>> may want to provide
> > > > >>>>>>>>>>>>>> him para5 runs.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia
for John
> > > > >>>>>>>>>>>>>> to do
> > > > >>>>>>>>>>>>>> some testing some production files. Do you
think I
> > > > >>>>>>>>>>>>>> should use parallels
> > > > >>>>>>>>>>>>>> instead or do you think the ops files are OK
for
> > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > >>>>>>>>>>>>>> for AQ?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the
ops files
> > > > >>>>>>>>>>>>>> should
> > > > >>>>>>>>>>>>>> be OK for now.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels.
> > > > >>>>>>>>>>>>>>> We
> > > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > > > >>>>>>>>>>>>>>> but
> > > > >>>>>>>>>>>>>>> management did not
> > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > > >>>>>>>>>>>>>>> Jeff
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
correct time
> > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send
John over
> > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I don't think that the MET team has any other
> > > > >>>>>>>>>>>>>>> versions
> > > > >>>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but
they can
> > > > >>>>>>>>>>>>>>> also clarify this as
> > > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen
<
> > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want
to use
> > > > >>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > > > >>>>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management
> > > > >>>>>>>>>>>>>>>> did
> > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > >>>>>>>>>>>>>>>> further delays so we will have to do so with
the
> > > > >>>>>>>>>>>>>>>> FYQ4
> > > > >>>>>>>>>>>>>>>> implementation.
> > > > >>>>>>>>>>>>>>>> Jeff
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > > > >>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation.
I
> > > > >>>>>>>>>>>>>>>> think
> > > > >>>>>>>>>>>>>>>> I
> > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some
time ago,
> > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
entirely
> > > > >>>>>>>>>>>>>>>> sure.
> > > > >>>>>>>>>>>>>>>> But he would have
> > > > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
> > > > >>>>>>>>>>>>>>>> approach
> > > > >>>>>>>>>>>>>>>> this.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues
John
> > > > >>>>>>>>>>>>>>>> talks
> > > > >>>>>>>>>>>>>>>> about?
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
> > > > >>>>>>>>>>>>>>>>> compute
> > > > >>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1-
and 8-
> > > > >>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
\
> > > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat
> \
> > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1, for
> > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > > >>>>>>>>>>>>>>>>> FULL,
> > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents
of
> > > > >>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>> "max"
> > > > >>>>>>>>>>>>>>>>> file:
> > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
level:17-40
> > > > >>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > > > >>>>>>>>>>>>>>>>> revealed
> > > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours 17
> > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> > > > >>>>>>>>>>>>>>>>> file, we
> > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it
which one
> > > > >>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > >>>>>>>>>>>>>>>>> field = [
> > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time
> > > > >>>>>>>>>>>>>>>>> =
> > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > >>>>>>>>>>>>>>>>> ];
> > > > >>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps
are off
> > > > >>>>>>>>>>>>>>>>> by 1
> > > > >>>>>>>>>>>>>>>>> hour.
> > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead
> > > > >>>>>>>>>>>>>>>>> time
> > > > >>>>>>>>>>>>>>>>> =
> > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> > > > >>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> > > > >>>>>>>>>>>>>>>>> file by
> > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > >>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > >>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries, but I
> > > > >>>>>>>>>>>>>>>>>> found
> > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on
dakota
> > > > >>>>>>>>>>>>>>>>>> here:
> > > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute 24-
> > > > >>>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count
> > > > >>>>>>>>>>>>>>>>>> up
> > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > >>>>>>>>>>>>>>>>>> wc
> > > > >>>>>>>>>>>>>>>>>> -l
> > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
pb2nc_output.txt | wc
> > > > >>>>>>>>>>>>>>>>>> -l
> > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > >>>>>>>>>>>>>>>>>> sort
> > > > >>>>>>>>>>>>>>>>>> -u |
> > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning
that
> > > > >>>>>>>>>>>>>>>>>> all
> > > > >>>>>>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > > > >>>>>>>>>>>>>>>>>> meaning
> > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that
file is
> > > > >>>>>>>>>>>>>>>>>> named
> > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated
in the
> > > > >>>>>>>>>>>>>>>>>> last two
> > > > >>>>>>>>>>>>>>>>>> lines.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> > > > >>>>>>>>>>>>>>>>>> are
> > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason
why
> > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > >>>>>>>>>>>>>>>>>>> have
> > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh
<
> > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check
in the
> > > > >>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content of
> > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
inputs. I
> > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files
and will
> > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change the
> > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard
Soh <
> > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the
logic for
> > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute
24-hour
> > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump
> > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in
PB2NC not
> > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in
the
> > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard Soh <
> > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00
and
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC. I
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600;
end = 0;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations...
but
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem
caused by
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of
them to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM
John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a
24
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next
week.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM
hsoh <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation
data in
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
without any
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the
NetCDF
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from
0 to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable. The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger
than 3655.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota in
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
pb2nc_output.txt
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists
all of
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But
some of
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced
by
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri May 03 10:06:11 2019
Hi, John,
Well, I am running the script in batch so I don't control what node it
gets. I can tell you that the working directory is here:
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
And the output log file is here:
/ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688, which
should
help you figure out how the thing is run.
Thanks!
Perry
On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry,
>
> We can log on to wcoss and test there instead. Yes, pb2nc does seem
to be
> the culprit. Can you point us the wcoss node you're using and the
working
> directory there? We basically need to recreate your call to pb2nc
to try
> to replicate the behavior.
>
> Thanks,
> John
>
> On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > Hi, David, John,
> >
> > I am not running on Theia. In fact I think it would take quite a
bit of
> > effort to transition this thing to Theia, and I'll need to be
running
> this
> > verification on prod data anyway, which is only on WCOSS.
> >
> > However, if you wish to troubleshoot on Theia, you should do so,
and I
> > think you have all you need to be able to do so, right?
> >
> > It would seem that the error on my side is in PB2NC, so I'll
probably
> need
> > to make sure I'm running what you're running.
> >
> > Thanks!
> >
> > Perry
> >
> > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Perry -
> > > Are you running on theia?
> > > John suggests that we try to troubleshoot there ...
> > > thanks,
> > > David
> > >
> > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > Hi Perry -
> > > >
> > > > An update on where we are on this issue:
> > > > Using the prepda.2019042312.nc file you provided we indeed
reproduce
> > > > the point_stat error.
> > > > The point_stat error is due to only a single entry the valid
time
> > > > table,
> > > > e.g. from ncdump -v hdr_vld_table
> > > > hdr_vld_table =
> > > > "20190423_050000",
> > > > while the hdr_vld indices in that file are hdr_vld = 0, 33,
33, ...
> > > > which are all out of range except the first.
> > > >
> > > > If I run pb2nc here with your PREPBUFR data and your
PB2NCConfig_aqm
> > > > config file, however,
> > > > I generate a different prepda.2019042312.nc file that runs
through
> to
> > > > completion with point_stat.
> > > > Its valid time table:
> > > >
> > > > hdr_vld_table =
> > > > "20190423_030000",
> > > > "20190423_010000",
> > > > "20190423_020000",
> > > > "20190423_230000",
> > > > "20190423_150000",
> > > > "20190423_090000",
> > > > "20190423_070000",
> > > > "20190423_060000",
> > > > "20190423_210000",
> > > > "20190423_050000",
> > > > "20190423_170000",
> > > > "20190423_100000",
> > > > "20190423_200000",
> > > > "20190423_140000",
> > > > "20190423_130000",
> > > > "20190423_160000",
> > > > "20190423_040000",
> > > > "20190423_180000",
> > > > "20190423_080000",
> > > > "20190423_120000",
> > > > "20190423_110000",
> > > > "20190423_220000",
> > > > "20190423_190000",
> > > > "20190424_000000" ;
> > > > }
> > > >
> > > > So it seems pb2nc is running correctly here, but not on your
side.
> > > > Let me discuss with John the best way to proceed ...
> > > > thanks,
> > > > David
> > > >
> > > >
> > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > Hi, everyone,
> > > > >
> > > > > I am attaching this to this long thread regarding the daily
max of
> > > > > ozone
> > > > > verification, so we remember what we were doing back then.
> > > > >
> > > > > I am now running this again using MET 8.0, and something
seems to
> not
> > > > > be
> > > > > working when I run point_stat. I get this error when
running a
> daily
> > > > > max
> > > > > of ozone:
> > > > >
> > > > > ERROR :
> > > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > > ERROR :
> > > > >
> > > > > I am using my own scripts to run this using MET, so this is
not
> using
> > > > > METplus.
> > > > >
> > > > > You can find a log file of this here:
> > > > >
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > It
> > > > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> > > > > fine, I
> > > > > believe.
> > > > >
> > > > > The script I am using to run this is here:
> > > > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > >
> > > > > The prepbufr files are:
> > > > >
> > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > >
> > > > > The model file is here:
> > > > >
> > > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > >
> > > > >
> > > > > What could be causing this error?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Perry
> > > > >
> > > > >
> > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> johnhg at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Great, glad that did the trick.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
Affiliate <
> > > > > > perry.shafran at noaa.gov> wrote:
> > > > > >
> > > > > >> Well this worked! And Jeff confirmed to me that the 8-hr
> average
> > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in
the 1-hr
> > > > > >> avg.
> > > > > >>
> > > > > >> Thanks!
> > > > > >>
> > > > > >> Perry
> > > > > >>
> > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > >> <johnhg at ucar.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Perry,
> > > > > >>>
> > > > > >>> I think the problem is in your setting of
> > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > >>>
> > > > > >>> Here's your wgrib2 output:
> > > > > >>> wgrib2 oz_DAY1
> > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> > > > > >>> fcst:
> > > > > >>>
> > > > > >>> The "d=2018081112" tells you the model initialization
time.
> The
> > > > > >>> valid
> > > > > >>> time = init time + forecast time. So the OZMAX1 data is
valid
> 16
> > > > > >>> hours
> > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> that
> > > > > >>> wgrib2
> > > > > >>> command using the "-vt" argument to list the
verification time.
> > > > > >>>
> > > > > >>> Point-Stat isn't finding any OZMAX8 data because that
data
> isn't
> > > > > >>> valid
> > > > > >>> at the time you're requesting.
> > > > > >>>
> > > > > >>> John
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate
> <
> > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > >>>
> > > > > >>>> Hi, John,
> > > > > >>>>
> > > > > >>>> I haven't been able to work on this for awhile, but now
I'm
> > > > > >>>> working on
> > > > > >>>> it again but seem to be running into issues.
> > > > > >>>>
> > > > > >>>> I have a file that has both the daily max for 1-hr
ozone and
> the
> > > > > >>>> daily
> > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > > > >>>>
> > > > > >>>> 9 (t10a2)
> > > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > >>>> >
> > > > > >>>> wgrib2 oz_DAY1
> > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> fcst:
> > > > > >>>> 10 (t10a2)
> > > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > >>>> >
> > > > > >>>>
> > > > > >>>> As you can see they have different forecast times. But
when I
> > > > > >>>> run
> > > > > >>>> point_stat, I get this output:
> > > > > >>>>
> > > > > >>>> DEBUG 2:
> > > > > >>>> DEBUG 2:
> > > > > >>>>
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > >>>> DEBUG 2:
> > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > >>>> WARNING:
> > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > > > >>>> records
> > > > > >>>> found for 'OZMAX8/L1'
> > > > > >>>> WARNING:
> > > > > >>>> WARNING:
> > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > >>>> OZMAX8/L1
> > > > > >>>> found in file: oz_DAY1
> > > > > >>>> WARNING:
> > > > > >>>>
> > > > > >>>> Here is the relevant portion of the point_stat config
file:
> > > > > >>>>
> > > > > >>>> fcst = {
> > > > > >>>> field = [
> > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > >>>> ];
> > > > > >>>> }
> > > > > >>>>
> > > > > >>>> obs = {
> > > > > >>>> convert(x) = x * 10^9;
> > > > > >>>> field = [
> > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > >>>> ];
> > > > > >>>> }
> > > > > >>>>
> > > > > >>>> I don't remember if I had this working previously for
both the
> > > > > >>>> 1hr
> > > > > >>>> and
> > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue for
> > > > > >>>> the
> > > > > >>>> 8hr
> > > > > >>>> ozone. Have any thoughts?
> > > > > >>>>
> > > > > >>>> Thanks!
> > > > > >>>>
> > > > > >>>> Perry
> > > > > >>>>
> > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
> Affiliate
> > > > > >>>> <
> > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>
> > > > > >>>>> I chose option 2, and that seems to work - I have a
.stat
> file
> > > > > >>>>> with
> > > > > >>>>> output. Now I'll need to do some work here to see if
the
> > > > > >>>>> output
> > > > > >>>>> looks OK,
> > > > > >>>>> and to configure the verification so it verifies
OZMAX1 and
> > > > > >>>>> OZMAX8.
> > > > > >>>>>
> > > > > >>>>> I also wonder if this would be easily modified for PM.
> > > > > >>>>>
> > > > > >>>>> Perry
> > > > > >>>>>
> > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > > > >>>>> <johnhg at ucar.edu>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Perry,
> > > > > >>>>>>
> > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > > > >>>>>> that
> > > > > >>>>>> are
> > > > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that
> are
> > > > > >>>>>> purportedly
> > > > > >>>>>> valid at 05Z. And that's the reason why you have so
many
> obs
> > > > > >>>>>> rejected due
> > > > > >>>>>> to the valid time.
> > > > > >>>>>>
> > > > > >>>>>> It's up to you guys to decide what the "correct" way
of
> > > > > >>>>>> handling
> > > > > >>>>>> this
> > > > > >>>>>> is... but here are a couple of options:
> > > > > >>>>>>
> > > > > >>>>>> (1) You could use the existing setup and address this
logic
> in
> > > > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > > > > >>>>>> from 0 to:
> > > > > >>>>>>
> > > > > >>>>>> obs_window = {
> > > > > >>>>>> beg = 3600;
> > > > > >>>>>> end = 3600;
> > > > > >>>>>> }
> > > > > >>>>>> This tells Point-Stat to match observations which are
> exactly
> > > > > >>>>>> 1-
> > > > > >>>>>> hour
> > > > > >>>>>> prior to the valid time of the forecast.
> > > > > >>>>>>
> > > > > >>>>>> (2) You could rerun PB2NC to compute a time summary
from 05Z
> > > > > >>>>>> to
> > > > > >>>>>> 05Z
> > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting at
> 0:
> > > > > >>>>>>
> > > > > >>>>>> time_summary = {
> > > > > >>>>>> flag = TRUE;
> > > > > >>>>>> beg = "050000";
> > > > > >>>>>> end = "050000";
> > > > > >>>>>> ...
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> John
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > > > > >>>>>> Affiliate
> > > > > >>>>>> <
> > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file, but
> > > > > >>>>>>> it's
> > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > >>>>>>>
> > > > > >>>>>>> DEBUG 4:
> > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> observation
> > > > > >>>>>>> type
> > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> method
> > > > > >>>>>>> NEAREST(1),
> > > > > >>>>>>> using 0 pairs.
> > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > >>>>>>>
> > > > > >>>>>>> Looks like most rejections were due to valid time.
> > > > > >>>>>>>
> > > > > >>>>>>> Have any thoughts about this one?
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks!
> > > > > >>>>>>>
> > > > > >>>>>>> Perry
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > >>>>>>> Affiliate
> > > > > >>>>>>> <
> > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Hi, John,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Looks like the changing of the time stamp from _04
to _05
> > > > > >>>>>>>> did
> > > > > >>>>>>>> the
> > > > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks!
> > > > > >>>>>>>>
> > > > > >>>>>>>> Perry
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Perry,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I agree, there's likely some issue with requesting
the
> > > > > >>>>>>>>> timestamp
> > > > > >>>>>>>>> of the data from that file. I think switching the
> > > > > >>>>>>>>> verification time to
> > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> initialization
> > > > > >>>>>>>>> times
> > > > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>> MET commands
> > > > > >>>>>>>>> to illustrate.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > > > >>>>>>>>> time
> > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using
a
> sample
> > > > > >>>>>>>>> data file you
> > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> wgrib2
> > > > > >>>>>>>>> so
> > > > > >>>>>>>>> I have that
> > > > > >>>>>>>>> funky forecast hour:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
verbosity
> > > > > >>>>>>>>> level
> > > > > >>>>>>>>> (-v
> > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data it is
> > > > > >>>>>>>>> reading:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> # Read the first record
> > > > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > >>>>>>>>> ...
> > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > >>>>>>>>> ...
> > > > > >>>>>>>>> # Read the second record
> > > > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > >>>>>>>>> ...
> > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Hope that helps.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks,
> > > > > >>>>>>>>> John
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran -
NOAA
> > > > > >>>>>>>>> Affiliate <
> > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Hi, John,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> When I implemented all the changes (and also set
it up
> so
> > > > > >>>>>>>>>> it's
> > > > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > >>>>>>>>>>
> > >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > >>>>>>>>>> -v 4
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> But then I got this error:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> DEBUG 2:
> > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
> > > > > >>>>>>>>>> matching
> > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
> matching
> > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> ERROR :
> > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > >>>>>>>>>> forecast
> > > > > >>>>>>>>>> data
> > > > > >>>>>>>>>> found! Exiting...
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output for
> the
> > > > > >>>>>>>>>> file
> > > > > >>>>>>>>>> in this case reads:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 14 (t14a2)
> > > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > >>>>>>>>>> 15 (t14a2)
> > > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You
had
> > > > > >>>>>>>>>> suggested
> > > > > >>>>>>>>>> that it should be the time stamp of the 2nd day,
but
> maybe
> > > > > >>>>>>>>>> it needs to be
> > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Perry
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Perry,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did,
> we
> > > > > >>>>>>>>>>> used
> > > > > >>>>>>>>>>> observation files named:
> > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> And those files don't exist in the July data you
just
> > > > > >>>>>>>>>>> sent.
> > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
aqm.t12z.anowpm.pb.tm00
> > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against the
> > > > > >>>>>>>>>>> BUFR
> > > > > >>>>>>>>>>> or
> > > > > >>>>>>>>>>> PREPBUFR.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Let me just tell you the key points for the
config file
> > > > > >>>>>>>>>>> settings...
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2
input
> > > > > >>>>>>>>>>> files
> > > > > >>>>>>>>>>> -
> > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > >>>>>>>>>>> pb2nc \
> > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > > > >>>>>>>>>>> like
> > > > > >>>>>>>>>>> this:
> > > > > >>>>>>>>>>> time_summary = {
> > > > > >>>>>>>>>>> flag = TRUE;
> > > > > >>>>>>>>>>> beg = "040000";
> > > > > >>>>>>>>>>> end = "040000";
> > > > > >>>>>>>>>>> step = 24*3600;
> > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; }
//
> > > > > >>>>>>>>>>> previous
> > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > >>>>>>>>>>> grib_code = [];
> > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every
hour
> > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries when
> at
> > > > > >>>>>>>>>>> least
> > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > >>>>>>>>>>> }
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > > > > >>>>>>>>>>> over the
> > > > > >>>>>>>>>>> defined time period. They're identified in the
output
> by
> > > > > >>>>>>>>>>> the message type:
> > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst"
and
> "obs"
> > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > >>>>>>>>>>> Instead
> > > > > >>>>>>>>>>> of specifying
> > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
> convenient
> > > > > >>>>>>>>>>> to specify the
> > > > > >>>>>>>>>>> valid time. So that you don't have to modify
the
> config
> > > > > >>>>>>>>>>> file, I'd suggest
> > > > > >>>>>>>>>>> using environment variables:
> > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> fcst = {
> > > > > >>>>>>>>>>> field = [
> > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > >>>>>>>>>>> ];
> > > > > >>>>>>>>>>> }
> > > > > >>>>>>>>>>> obs = {
> > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > >>>>>>>>>>> field = [
> > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > >>>>>>>>>>> ];
> > > > > >>>>>>>>>>> }
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway <
> > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > > > >>>>>>>>>>>> Affiliate <
> > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc
grids and
> > > > > >>>>>>>>>>>>> focus on
> > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files
> > > > > >>>>>>>>>>>>> over
> > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > >>>>>>>>>>>>> let him work on the files he has for now and
if he
> runs
> > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least the
24-hr
> > > > > >>>>>>>>>>>>> max/avg files).
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks!
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not for
> > > > > >>>>>>>>>>>>>> bias
> > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe,
> > > > > >>>>>>>>>>>>>> you
> > > > > >>>>>>>>>>>>>> may want to provide
> > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia
for
> John
> > > > > >>>>>>>>>>>>>> to do
> > > > > >>>>>>>>>>>>>> some testing some production files. Do you
think I
> > > > > >>>>>>>>>>>>>> should use parallels
> > > > > >>>>>>>>>>>>>> instead or do you think the ops files are OK
for
> > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > >>>>>>>>>>>>>> for AQ?
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the
ops
> files
> > > > > >>>>>>>>>>>>>> should
> > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen
<
> > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels.
> > > > > >>>>>>>>>>>>>>> We
> > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > > > > >>>>>>>>>>>>>>> but
> > > > > >>>>>>>>>>>>>>> management did not
> > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > > > >>>>>>>>>>>>>>> Jeff
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
> Affiliate
> > > > > >>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
correct
> time
> > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send
John
> over
> > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > >>>>>>>>>>>>>>> versions
> > > > > >>>>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but
they
> can
> > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new
does over
> > > > > >>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want to
> use
> > > > > >>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > > > > >>>>>>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management
> > > > > >>>>>>>>>>>>>>>> did
> > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with the
> > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
> Affiliate
> > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation. I
> > > > > >>>>>>>>>>>>>>>> think
> > > > > >>>>>>>>>>>>>>>> I
> > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some
time
> ago,
> > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff.
I
> think
> > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
entirely
> > > > > >>>>>>>>>>>>>>>> sure.
> > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and how
to
> > > > > >>>>>>>>>>>>>>>> approach
> > > > > >>>>>>>>>>>>>>>> this.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues John
> > > > > >>>>>>>>>>>>>>>> talks
> > > > > >>>>>>>>>>>>>>>> about?
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
> Gotway <
> > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away
to
> > > > > >>>>>>>>>>>>>>>>> compute
> > > > > >>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1-
and 8-
> > > > > >>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > >>>>>>>>>>>>>>>>> -pbfile
> aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > \
> > > > > >>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1, for
> > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents of
> > > > > >>>>>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > >>>>>>>>>>>>>>>>> file:
> > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> level:17-40
> > > > > >>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours
> 17
> > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in
the same
> > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it
which one
> > > > > >>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> lead_time
> > > > > >>>>>>>>>>>>>>>>> =
> > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > >>>>>>>>>>>>>>>>> ];
> > > > > >>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps are
> off
> > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead
> > > > > >>>>>>>>>>>>>>>>> time
> > > > > >>>>>>>>>>>>>>>>> =
> > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour
max
> value
> > > > > >>>>>>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at
05Z
> while
> > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-
Stat
> config
> > > > > >>>>>>>>>>>>>>>>> file by
> > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > >>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> but I
> > > > > >>>>>>>>>>>>>>>>>> found
> > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on
dakota
> > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute 24-
> > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped
the
> output
> > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands
to count
> > > > > >>>>>>>>>>>>>>>>>> up
> > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt
> |
> > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt
> |
> > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> wc
> > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning that
> > > > > >>>>>>>>>>>>>>>>>> all
> > > > > >>>>>>>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations
are
> > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that
file is
> > > > > >>>>>>>>>>>>>>>>>> named
> > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated
in the
> > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
> duplicates
> > > > > >>>>>>>>>>>>>>>>>> are
> > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason why
> > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard
Soh <
> > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
> (550.319
> > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check in
> the
> > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content
> of
> > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
inputs. I
> > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files
and
> will
> > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change the
> > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard
Soh <
> > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the
logic
> for
> > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> 24-hour
> > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
> counts:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript
to dump
> > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in
PB2NC not
> > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or
in the
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> Soh <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00 and
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM
John
> Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC.
> I
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600;
end =
> 0;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations...
but
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what you'd
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> "AIRNOW_MAX_240000"
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem
caused by
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2
of them
> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM
John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute
a 24
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next
week.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM
hsoh <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation
data in
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file
was
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
without
> any
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the
NetCDF
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from
0 to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
> summary)
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable. The
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger
than
> 3655.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> Gotway
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota
> in
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it
to
> ascii:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
pb2nc_output.txt
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists
all of
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But
some of
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced by
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> pb2nc_output.nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>
> > >
> > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Mon Jun 03 12:16:16 2019
Perry,
Did you ever get this issue resolved from 1 month ago? Or should I
log on
to WCOSS and take a look?
Thanks,
John
On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
> Hi, John,
>
> Well, I am running the script in batch so I don't control what node
it
> gets. I can tell you that the working directory is here:
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>
> And the output log file is here:
> /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688, which
should
> help you figure out how the thing is run.
>
> Thanks!
>
> Perry
>
> On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Perry,
> >
> > We can log on to wcoss and test there instead. Yes, pb2nc does
seem to
> be
> > the culprit. Can you point us the wcoss node you're using and the
> working
> > directory there? We basically need to recreate your call to pb2nc
to try
> > to replicate the behavior.
> >
> > Thanks,
> > John
> >
> > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >
> > > Hi, David, John,
> > >
> > > I am not running on Theia. In fact I think it would take quite
a bit
> of
> > > effort to transition this thing to Theia, and I'll need to be
running
> > this
> > > verification on prod data anyway, which is only on WCOSS.
> > >
> > > However, if you wish to troubleshoot on Theia, you should do so,
and I
> > > think you have all you need to be able to do so, right?
> > >
> > > It would seem that the error on my side is in PB2NC, so I'll
probably
> > need
> > > to make sure I'm running what you're running.
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Perry -
> > > > Are you running on theia?
> > > > John suggests that we try to troubleshoot there ...
> > > > thanks,
> > > > David
> > > >
> > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > Hi Perry -
> > > > >
> > > > > An update on where we are on this issue:
> > > > > Using the prepda.2019042312.nc file you provided we indeed
> reproduce
> > > > > the point_stat error.
> > > > > The point_stat error is due to only a single entry the valid
time
> > > > > table,
> > > > > e.g. from ncdump -v hdr_vld_table
> > > > > hdr_vld_table =
> > > > > "20190423_050000",
> > > > > while the hdr_vld indices in that file are hdr_vld = 0, 33,
33, ...
> > > > > which are all out of range except the first.
> > > > >
> > > > > If I run pb2nc here with your PREPBUFR data and your
> PB2NCConfig_aqm
> > > > > config file, however,
> > > > > I generate a different prepda.2019042312.nc file that runs
through
> > to
> > > > > completion with point_stat.
> > > > > Its valid time table:
> > > > >
> > > > > hdr_vld_table =
> > > > > "20190423_030000",
> > > > > "20190423_010000",
> > > > > "20190423_020000",
> > > > > "20190423_230000",
> > > > > "20190423_150000",
> > > > > "20190423_090000",
> > > > > "20190423_070000",
> > > > > "20190423_060000",
> > > > > "20190423_210000",
> > > > > "20190423_050000",
> > > > > "20190423_170000",
> > > > > "20190423_100000",
> > > > > "20190423_200000",
> > > > > "20190423_140000",
> > > > > "20190423_130000",
> > > > > "20190423_160000",
> > > > > "20190423_040000",
> > > > > "20190423_180000",
> > > > > "20190423_080000",
> > > > > "20190423_120000",
> > > > > "20190423_110000",
> > > > > "20190423_220000",
> > > > > "20190423_190000",
> > > > > "20190424_000000" ;
> > > > > }
> > > > >
> > > > > So it seems pb2nc is running correctly here, but not on your
side.
> > > > > Let me discuss with John the best way to proceed ...
> > > > > thanks,
> > > > > David
> > > > >
> > > > >
> > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > > Hi, everyone,
> > > > > >
> > > > > > I am attaching this to this long thread regarding the
daily max
> of
> > > > > > ozone
> > > > > > verification, so we remember what we were doing back then.
> > > > > >
> > > > > > I am now running this again using MET 8.0, and something
seems to
> > not
> > > > > > be
> > > > > > working when I run point_stat. I get this error when
running a
> > daily
> > > > > > max
> > > > > > of ozone:
> > > > > >
> > > > > > ERROR :
> > > > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > > > ERROR :
> > > > > >
> > > > > > I am using my own scripts to run this using MET, so this
is not
> > using
> > > > > > METplus.
> > > > > >
> > > > > > You can find a log file of this here:
> > > > > >
> > >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > It
> > > > > > runs pb2nc and point_stat for max ozone. The pb2nc code is
> working
> > > > > > fine, I
> > > > > > believe.
> > > > > >
> > > > > > The script I am using to run this is here:
> > > > > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > >
> > > > > > The prepbufr files are:
> > > > > >
> > > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
and
> > > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > >
> > > > > > The model file is here:
> > > > > >
> > > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > >
> > > > > >
> > > > > > What could be causing this error?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Perry
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > johnhg at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Great, glad that did the trick.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
Affiliate
> <
> > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > >
> > > > > > >> Well this worked! And Jeff confirmed to me that the 8-
hr
> > average
> > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in
the
> 1-hr
> > > > > > >> avg.
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >> Perry
> > > > > > >>
> > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > >> <johnhg at ucar.edu>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Perry,
> > > > > > >>>
> > > > > > >>> I think the problem is in your setting of
> > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > >>>
> > > > > > >>> Here's your wgrib2 output:
> > > > > > >>> wgrib2 oz_DAY1
> > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour ave
> > > > > > >>> fcst:
> > > > > > >>>
> > > > > > >>> The "d=2018081112" tells you the model initialization
time.
> > The
> > > > > > >>> valid
> > > > > > >>> time = init time + forecast time. So the OZMAX1 data
is
> valid
> > 16
> > > > > > >>> hours
> > > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> > that
> > > > > > >>> wgrib2
> > > > > > >>> command using the "-vt" argument to list the
verification
> time.
> > > > > > >>>
> > > > > > >>> Point-Stat isn't finding any OZMAX8 data because that
data
> > isn't
> > > > > > >>> valid
> > > > > > >>> at the time you're requesting.
> > > > > > >>>
> > > > > > >>> John
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
> Affiliate
> > <
> > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>
> > > > > > >>>> Hi, John,
> > > > > > >>>>
> > > > > > >>>> I haven't been able to work on this for awhile, but
now I'm
> > > > > > >>>> working on
> > > > > > >>>> it again but seem to be running into issues.
> > > > > > >>>>
> > > > > > >>>> I have a file that has both the daily max for 1-hr
ozone and
> > the
> > > > > > >>>> daily
> > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > >>>>
> > > > > > >>>> 9 (t10a2)
> > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > >>>> >
> > > > > > >>>> wgrib2 oz_DAY1
> > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> > fcst:
> > > > > > >>>> 10 (t10a2)
> > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > >>>> >
> > > > > > >>>>
> > > > > > >>>> As you can see they have different forecast times.
But
> when I
> > > > > > >>>> run
> > > > > > >>>> point_stat, I get this output:
> > > > > > >>>>
> > > > > > >>>> DEBUG 2:
> > > > > > >>>> DEBUG 2:
> > > > > > >>>>
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > >>>> DEBUG 2:
> > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > >>>> WARNING:
> > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > > > > >>>> records
> > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > >>>> WARNING:
> > > > > > >>>> WARNING:
> > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > > >>>> OZMAX8/L1
> > > > > > >>>> found in file: oz_DAY1
> > > > > > >>>> WARNING:
> > > > > > >>>>
> > > > > > >>>> Here is the relevant portion of the point_stat config
file:
> > > > > > >>>>
> > > > > > >>>> fcst = {
> > > > > > >>>> field = [
> > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > >>>> ];
> > > > > > >>>> }
> > > > > > >>>>
> > > > > > >>>> obs = {
> > > > > > >>>> convert(x) = x * 10^9;
> > > > > > >>>> field = [
> > > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > >>>> ];
> > > > > > >>>> }
> > > > > > >>>>
> > > > > > >>>> I don't remember if I had this working previously for
both
> the
> > > > > > >>>> 1hr
> > > > > > >>>> and
> > > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue
> for
> > > > > > >>>> the
> > > > > > >>>> 8hr
> > > > > > >>>> ozone. Have any thoughts?
> > > > > > >>>>
> > > > > > >>>> Thanks!
> > > > > > >>>>
> > > > > > >>>> Perry
> > > > > > >>>>
> > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > >>>> <
> > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>
> > > > > > >>>>> I chose option 2, and that seems to work - I have a
.stat
> > file
> > > > > > >>>>> with
> > > > > > >>>>> output. Now I'll need to do some work here to see
if the
> > > > > > >>>>> output
> > > > > > >>>>> looks OK,
> > > > > > >>>>> and to configure the verification so it verifies
OZMAX1 and
> > > > > > >>>>> OZMAX8.
> > > > > > >>>>>
> > > > > > >>>>> I also wonder if this would be easily modified for
PM.
> > > > > > >>>>>
> > > > > > >>>>> Perry
> > > > > > >>>>>
> > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Perry,
> > > > > > >>>>>>
> > > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > > > > >>>>>> that
> > > > > > >>>>>> are
> > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts that
> > are
> > > > > > >>>>>> purportedly
> > > > > > >>>>>> valid at 05Z. And that's the reason why you have
so many
> > obs
> > > > > > >>>>>> rejected due
> > > > > > >>>>>> to the valid time.
> > > > > > >>>>>>
> > > > > > >>>>>> It's up to you guys to decide what the "correct"
way of
> > > > > > >>>>>> handling
> > > > > > >>>>>> this
> > > > > > >>>>>> is... but here are a couple of options:
> > > > > > >>>>>>
> > > > > > >>>>>> (1) You could use the existing setup and address
this
> logic
> > in
> > > > > > >>>>>> Point-Stat config file. Change the matching time
> obs_window
> > > > > > >>>>>> from 0 to:
> > > > > > >>>>>>
> > > > > > >>>>>> obs_window = {
> > > > > > >>>>>> beg = 3600;
> > > > > > >>>>>> end = 3600;
> > > > > > >>>>>> }
> > > > > > >>>>>> This tells Point-Stat to match observations which
are
> > exactly
> > > > > > >>>>>> 1-
> > > > > > >>>>>> hour
> > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > >>>>>>
> > > > > > >>>>>> (2) You could rerun PB2NC to compute a time summary
from
> 05Z
> > > > > > >>>>>> to
> > > > > > >>>>>> 05Z
> > > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting
> at
> > 0:
> > > > > > >>>>>>
> > > > > > >>>>>> time_summary = {
> > > > > > >>>>>> flag = TRUE;
> > > > > > >>>>>> beg = "050000";
> > > > > > >>>>>> end = "050000";
> > > > > > >>>>>> ...
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks,
> > > > > > >>>>>> John
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran -
NOAA
> > > > > > >>>>>> Affiliate
> > > > > > >>>>>> <
> > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file,
> but
> > > > > > >>>>>>> it's
> > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > >>>>>>>
> > > > > > >>>>>>> DEBUG 4:
> > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> > observation
> > > > > > >>>>>>> type
> > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> > method
> > > > > > >>>>>>> NEAREST(1),
> > > > > > >>>>>>> using 0 pairs.
> > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > >>>>>>>
> > > > > > >>>>>>> Looks like most rejections were due to valid time.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks!
> > > > > > >>>>>>>
> > > > > > >>>>>>> Perry
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > > >>>>>>> Affiliate
> > > > > > >>>>>>> <
> > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Hi, John,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Looks like the changing of the time stamp from
_04 to
> _05
> > > > > > >>>>>>>> did
> > > > > > >>>>>>>> the
> > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> change.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks!
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Perry
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Perry,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting the
> > > > > > >>>>>>>>> timestamp
> > > > > > >>>>>>>>> of the data from that file. I think switching
the
> > > > > > >>>>>>>>> verification time to
> > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > initialization
> > > > > > >>>>>>>>> times
> > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> wgrib2
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>> MET commands
> > > > > > >>>>>>>>> to illustrate.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > > > > >>>>>>>>> time
> > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > sample
> > > > > > >>>>>>>>> data file you
> > > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> > wgrib2
> > > > > > >>>>>>>>> so
> > > > > > >>>>>>>>> I have that
> > > > > > >>>>>>>>> funky forecast hour:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour
ave
> > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
verbosity
> > > > > > >>>>>>>>> level
> > > > > > >>>>>>>>> (-v
> > > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data it
> is
> > > > > > >>>>>>>>> reading:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> # Read the first record
> > > > > > >>>>>>>>> plot_data_plane
> > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > >>>>>>>>> ...
> > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > >>>>>>>>> ...
> > > > > > >>>>>>>>> # Read the second record
> > > > > > >>>>>>>>> plot_data_plane
> > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > >>>>>>>>> ...
> > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Hope that helps.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks,
> > > > > > >>>>>>>>> John
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran -
NOAA
> > > > > > >>>>>>>>> Affiliate <
> > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Hi, John,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> When I implemented all the changes (and also
set it up
> > so
> > > > > > >>>>>>>>>> it's
> > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > >>>>>>>>>>
> > > >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > >>>>>>>>>> -v 4
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> But then I got this error:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
> > > > > > >>>>>>>>>> matching
> > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > matching
> > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> ERROR :
> > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > > >>>>>>>>>> forecast
> > > > > > >>>>>>>>>> data
> > > > > > >>>>>>>>>> found! Exiting...
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output for
> > the
> > > > > > >>>>>>>>>> file
> > > > > > >>>>>>>>>> in this case reads:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > >>>>>>>>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >
> > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour ave
> > fcst:
> > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > >>>>>>>>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You
had
> > > > > > >>>>>>>>>> suggested
> > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day, but
> > maybe
> > > > > > >>>>>>>>>> it needs to be
> > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Perry
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Perry,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I'm confused. In the previous air quality vx
we did,
> > we
> > > > > > >>>>>>>>>>> used
> > > > > > >>>>>>>>>>> observation files named:
> > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> And those files don't exist in the July data
you just
> > > > > > >>>>>>>>>>> sent.
> > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
aqm.t12z.anowpm.pb.tm00
> > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against the
> > > > > > >>>>>>>>>>> BUFR
> > > > > > >>>>>>>>>>> or
> > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Let me just tell you the key points for the
config
> file
> > > > > > >>>>>>>>>>> settings...
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2
input
> > > > > > >>>>>>>>>>> files
> > > > > > >>>>>>>>>>> -
> > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > >>>>>>>>>>> pb2nc \
> > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > > > > >>>>>>>>>>> like
> > > > > > >>>>>>>>>>> this:
> > > > > > >>>>>>>>>>> time_summary = {
> > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > >>>>>>>>>>> end = "040000";
> > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; }
//
> > > > > > >>>>>>>>>>> previous
> > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every
hour
> > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> when
> > at
> > > > > > >>>>>>>>>>> least
> > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > >>>>>>>>>>> }
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> station
> > > > > > >>>>>>>>>>> over the
> > > > > > >>>>>>>>>>> defined time period. They're identified in
the
> output
> > by
> > > > > > >>>>>>>>>>> the message type:
> > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst" and
> > "obs"
> > > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > > >>>>>>>>>>> Instead
> > > > > > >>>>>>>>>>> of specifying
> > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
> > convenient
> > > > > > >>>>>>>>>>> to specify the
> > > > > > >>>>>>>>>>> valid time. So that you don't have to modify
the
> > config
> > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > >>>>>>>>>>> using environment variables:
> > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> fcst = {
> > > > > > >>>>>>>>>>> field = [
> > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > >>>>>>>>>>> ];
> > > > > > >>>>>>>>>>> }
> > > > > > >>>>>>>>>>> obs = {
> > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > >>>>>>>>>>> field = [
> > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > >>>>>>>>>>> ];
> > > > > > >>>>>>>>>>> }
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway <
> > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran
- NOAA
> > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc
grids
> and
> > > > > > >>>>>>>>>>>>> focus on
> > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy files
> > > > > > >>>>>>>>>>>>> over
> > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > >>>>>>>>>>>>> let him work on the files he has for now and
if he
> > runs
> > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least
the 24-hr
> > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen
<
> > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not
> for
> > > > > > >>>>>>>>>>>>>> bias
> > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To
be
> safe,
> > > > > > >>>>>>>>>>>>>> you
> > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
> Affiliate
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia for
> > John
> > > > > > >>>>>>>>>>>>>> to do
> > > > > > >>>>>>>>>>>>>> some testing some production files. Do you
think
> I
> > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > >>>>>>>>>>>>>> instead or do you think the ops files are
OK for
> > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that
the ops
> > files
> > > > > > >>>>>>>>>>>>>> should
> > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
> parallels.
> > > > > > >>>>>>>>>>>>>>> We
> > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max
hour
> labels
> > > > > > >>>>>>>>>>>>>>> but
> > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
correct
> > time
> > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send
John
> > over
> > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > > >>>>>>>>>>>>>>> versions
> > > > > > >>>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have -
but they
> > can
> > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new
does
> over
> > > > > > >>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want to
> > use
> > > > > > >>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
> Jianping
> > > > > > >>>>>>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
> management
> > > > > > >>>>>>>>>>>>>>>> did
> > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with the
> > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
> > Affiliate
> > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation. I
> > > > > > >>>>>>>>>>>>>>>> think
> > > > > > >>>>>>>>>>>>>>>> I
> > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some time
> > ago,
> > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff. I
> > think
> > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
entirely
> > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and
how to
> > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> John
> > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > Gotway <
> > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away to
> > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for
1- and
> 8-
> > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>>
> > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > \
> > > > > > >>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1,
> for
> > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents of
> > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > >>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> > level:17-40
> > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
> testing
> > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> hours
> > 17
> > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
> better.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in
the
> same
> > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it
which
> one
> > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > lead_time
> > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps are
> > off
> > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41
hour lead
> > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour
max
> > value
> > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at
05Z
> > while
> > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-
Stat
> > config
> > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and
was
> fixed.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> > but I
> > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on
dakota
> > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > >>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute
> 24-
> > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped
the
> > output
> > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands
to
> count
> > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> pb2nc_output.txt
> > |
> > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> pb2nc_output.txt
> > |
> > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > wc
> > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> that
> > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> 6,555
> > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations
are
> > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that file
> is
> > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt
| grep
> > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated in
> the
> > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
> > duplicates
> > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason why
> > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
> across 2
> > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which
we could
> > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard
Soh <
> > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > (550.319
> > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check in
> > the
> > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content
> > of
> > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
> summary
> > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
inputs.
> I
> > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files and
> > will
> > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> > wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change
> the
> > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
> currently
> > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard Soh
> <
> > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
> summary
> > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the
logic
> > for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > 24-hour
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > counts:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript to
> dump
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> string
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in
PB2NC
> not
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or
in the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look
in
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM
John
> Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> > Soh <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> and
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the
obs
> data.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM
John
> > Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could
you
> please
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC.
> > I
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request
> 24-hourly
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600; end =
> > 0;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations... but
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what
> you'd
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > "AIRNOW_MAX_240000"
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem
caused
> by
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2
of
> them
> > to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM
John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
> PrepBUFR
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute a 24
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next
week.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40
PM hsoh
> <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
> reasonable
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation data
> in
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file
was
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
without
> > any
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
> headers
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the
NetCDF
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> currently)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value
from 0
> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type
for
> > summary)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable.
> The
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger
than
> > 3655.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be
OK.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> > Gotway
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota
> > in
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump
it to
> > ascii:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
pb2nc_output.txt
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists all
> of
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But
some
> of
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced by
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > pb2nc_output.nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing
or table
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Mon Jun 03 12:57:27 2019
Hi, John,
To be honest, I haven't gotten a chance to look at it. Now that MET
8.1 is
up, I will try it again in the next day or two and let you know.
Thanks for reminding me!
Perry
On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry,
>
> Did you ever get this issue resolved from 1 month ago? Or should I
log on
> to WCOSS and take a look?
>
> Thanks,
> John
>
> On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> > Hi, John,
> >
> > Well, I am running the script in batch so I don't control what
node it
> > gets. I can tell you that the working directory is here:
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > And the output log file is here:
> > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
> should
> > help you figure out how the thing is run.
> >
> > Thanks!
> >
> > Perry
> >
> > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Perry,
> > >
> > > We can log on to wcoss and test there instead. Yes, pb2nc does
seem to
> > be
> > > the culprit. Can you point us the wcoss node you're using and
the
> > working
> > > directory there? We basically need to recreate your call to
pb2nc to
> try
> > > to replicate the behavior.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > Hi, David, John,
> > > >
> > > > I am not running on Theia. In fact I think it would take
quite a bit
> > of
> > > > effort to transition this thing to Theia, and I'll need to be
running
> > > this
> > > > verification on prod data anyway, which is only on WCOSS.
> > > >
> > > > However, if you wish to troubleshoot on Theia, you should do
so, and
> I
> > > > think you have all you need to be able to do so, right?
> > > >
> > > > It would seem that the error on my side is in PB2NC, so I'll
probably
> > > need
> > > > to make sure I'm running what you're running.
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Perry -
> > > > > Are you running on theia?
> > > > > John suggests that we try to troubleshoot there ...
> > > > > thanks,
> > > > > David
> > > > >
> > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > > Hi Perry -
> > > > > >
> > > > > > An update on where we are on this issue:
> > > > > > Using the prepda.2019042312.nc file you provided we indeed
> > reproduce
> > > > > > the point_stat error.
> > > > > > The point_stat error is due to only a single entry the
valid time
> > > > > > table,
> > > > > > e.g. from ncdump -v hdr_vld_table
> > > > > > hdr_vld_table =
> > > > > > "20190423_050000",
> > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33, 33,
> ...
> > > > > > which are all out of range except the first.
> > > > > >
> > > > > > If I run pb2nc here with your PREPBUFR data and your
> > PB2NCConfig_aqm
> > > > > > config file, however,
> > > > > > I generate a different prepda.2019042312.nc file that runs
> through
> > > to
> > > > > > completion with point_stat.
> > > > > > Its valid time table:
> > > > > >
> > > > > > hdr_vld_table =
> > > > > > "20190423_030000",
> > > > > > "20190423_010000",
> > > > > > "20190423_020000",
> > > > > > "20190423_230000",
> > > > > > "20190423_150000",
> > > > > > "20190423_090000",
> > > > > > "20190423_070000",
> > > > > > "20190423_060000",
> > > > > > "20190423_210000",
> > > > > > "20190423_050000",
> > > > > > "20190423_170000",
> > > > > > "20190423_100000",
> > > > > > "20190423_200000",
> > > > > > "20190423_140000",
> > > > > > "20190423_130000",
> > > > > > "20190423_160000",
> > > > > > "20190423_040000",
> > > > > > "20190423_180000",
> > > > > > "20190423_080000",
> > > > > > "20190423_120000",
> > > > > > "20190423_110000",
> > > > > > "20190423_220000",
> > > > > > "20190423_190000",
> > > > > > "20190424_000000" ;
> > > > > > }
> > > > > >
> > > > > > So it seems pb2nc is running correctly here, but not on
your
> side.
> > > > > > Let me discuss with John the best way to proceed ...
> > > > > > thanks,
> > > > > > David
> > > > > >
> > > > > >
> > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > > > Hi, everyone,
> > > > > > >
> > > > > > > I am attaching this to this long thread regarding the
daily max
> > of
> > > > > > > ozone
> > > > > > > verification, so we remember what we were doing back
then.
> > > > > > >
> > > > > > > I am now running this again using MET 8.0, and something
seems
> to
> > > not
> > > > > > > be
> > > > > > > working when I run point_stat. I get this error when
running a
> > > daily
> > > > > > > max
> > > > > > > of ozone:
> > > > > > >
> > > > > > > ERROR :
> > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> error!
> > > > > > > ERROR :
> > > > > > >
> > > > > > > I am using my own scripts to run this using MET, so this
is not
> > > using
> > > > > > > METplus.
> > > > > > >
> > > > > > > You can find a log file of this here:
> > > > > > >
> > > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > > It
> > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc code
is
> > working
> > > > > > > fine, I
> > > > > > > believe.
> > > > > > >
> > > > > > > The script I am using to run this is here:
> > > > > > >
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > > >
> > > > > > > The prepbufr files are:
> > > > > > >
> > > > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
and
> > > > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > > >
> > > > > > > The model file is here:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > > >
> > > > > > >
> > > > > > > What could be causing this error?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Perry
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > > johnhg at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Great, glad that did the trick.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
> Affiliate
> > <
> > > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > > >
> > > > > > > >> Well this worked! And Jeff confirmed to me that the
8-hr
> > > average
> > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like
in the
> > 1-hr
> > > > > > > >> avg.
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> Perry
> > > > > > > >>
> > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > > >> <johnhg at ucar.edu>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Perry,
> > > > > > > >>>
> > > > > > > >>> I think the problem is in your setting of
> > > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > > >>>
> > > > > > > >>> Here's your wgrib2 output:
> > > > > > > >>> wgrib2 oz_DAY1
> > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
> ave
> > > > > > > >>> fcst:
> > > > > > > >>>
> > > > > > > >>> The "d=2018081112" tells you the model
initialization time.
> > > The
> > > > > > > >>> valid
> > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
> > valid
> > > 16
> > > > > > > >>> hours
> > > > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> > > that
> > > > > > > >>> wgrib2
> > > > > > > >>> command using the "-vt" argument to list the
verification
> > time.
> > > > > > > >>>
> > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that data
> > > isn't
> > > > > > > >>> valid
> > > > > > > >>> at the time you're requesting.
> > > > > > > >>>
> > > > > > > >>> John
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
> > Affiliate
> > > <
> > > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi, John,
> > > > > > > >>>>
> > > > > > > >>>> I haven't been able to work on this for awhile, but
now
> I'm
> > > > > > > >>>> working on
> > > > > > > >>>> it again but seem to be running into issues.
> > > > > > > >>>>
> > > > > > > >>>> I have a file that has both the daily max for 1-hr
ozone
> and
> > > the
> > > > > > > >>>> daily
> > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > > >>>>
> > > > > > > >>>> 9 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>> wgrib2 oz_DAY1
> > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour ave
> > > fcst:
> > > > > > > >>>> 10 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>>
> > > > > > > >>>> As you can see they have different forecast times.
But
> > when I
> > > > > > > >>>> run
> > > > > > > >>>> point_stat, I get this output:
> > > > > > > >>>>
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>>
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> matching
> > > > > > > >>>> records
> > > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > > > >>>> OZMAX8/L1
> > > > > > > >>>> found in file: oz_DAY1
> > > > > > > >>>> WARNING:
> > > > > > > >>>>
> > > > > > > >>>> Here is the relevant portion of the point_stat
config
> file:
> > > > > > > >>>>
> > > > > > > >>>> fcst = {
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> obs = {
> > > > > > > >>>> convert(x) = x * 10^9;
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> I don't remember if I had this working previously
for both
> > the
> > > > > > > >>>> 1hr
> > > > > > > >>>> and
> > > > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue
> > for
> > > > > > > >>>> the
> > > > > > > >>>> 8hr
> > > > > > > >>>> ozone. Have any thoughts?
> > > > > > > >>>>
> > > > > > > >>>> Thanks!
> > > > > > > >>>>
> > > > > > > >>>> Perry
> > > > > > > >>>>
> > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>> <
> > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> I chose option 2, and that seems to work - I have
a .stat
> > > file
> > > > > > > >>>>> with
> > > > > > > >>>>> output. Now I'll need to do some work here to see
if the
> > > > > > > >>>>> output
> > > > > > > >>>>> looks OK,
> > > > > > > >>>>> and to configure the verification so it verifies
OZMAX1
> and
> > > > > > > >>>>> OZMAX8.
> > > > > > > >>>>>
> > > > > > > >>>>> I also wonder if this would be easily modified for
PM.
> > > > > > > >>>>>
> > > > > > > >>>>> Perry
> > > > > > > >>>>>
> > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> > > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Perry,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
> summaries
> > > > > > > >>>>>> that
> > > > > > > >>>>>> are
> > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
> that
> > > are
> > > > > > > >>>>>> purportedly
> > > > > > > >>>>>> valid at 05Z. And that's the reason why you have
so
> many
> > > obs
> > > > > > > >>>>>> rejected due
> > > > > > > >>>>>> to the valid time.
> > > > > > > >>>>>>
> > > > > > > >>>>>> It's up to you guys to decide what the "correct"
way of
> > > > > > > >>>>>> handling
> > > > > > > >>>>>> this
> > > > > > > >>>>>> is... but here are a couple of options:
> > > > > > > >>>>>>
> > > > > > > >>>>>> (1) You could use the existing setup and address
this
> > logic
> > > in
> > > > > > > >>>>>> Point-Stat config file. Change the matching time
> > obs_window
> > > > > > > >>>>>> from 0 to:
> > > > > > > >>>>>>
> > > > > > > >>>>>> obs_window = {
> > > > > > > >>>>>> beg = 3600;
> > > > > > > >>>>>> end = 3600;
> > > > > > > >>>>>> }
> > > > > > > >>>>>> This tells Point-Stat to match observations which
are
> > > exactly
> > > > > > > >>>>>> 1-
> > > > > > > >>>>>> hour
> > > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > > >>>>>>
> > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary from
> > 05Z
> > > > > > > >>>>>> to
> > > > > > > >>>>>> 05Z
> > > > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting
> > at
> > > 0:
> > > > > > > >>>>>>
> > > > > > > >>>>>> time_summary = {
> > > > > > > >>>>>> flag = TRUE;
> > > > > > > >>>>>> beg = "050000";
> > > > > > > >>>>>> end = "050000";
> > > > > > > >>>>>> ...
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks,
> > > > > > > >>>>>> John
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran -
NOAA
> > > > > > > >>>>>> Affiliate
> > > > > > > >>>>>> <
> > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file,
> > but
> > > > > > > >>>>>>> it's
> > > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 4:
> > > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > observation
> > > > > > > >>>>>>> type
> > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> > > method
> > > > > > > >>>>>>> NEAREST(1),
> > > > > > > >>>>>>> using 0 pairs.
> > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thanks!
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Perry
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > > > >>>>>>> Affiliate
> > > > > > > >>>>>>> <
> > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Hi, John,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Looks like the changing of the time stamp from
_04 to
> > _05
> > > > > > > >>>>>>>> did
> > > > > > > >>>>>>>> the
> > > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> > change.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks!
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Perry
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Perry,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
> the
> > > > > > > >>>>>>>>> timestamp
> > > > > > > >>>>>>>>> of the data from that file. I think switching
the
> > > > > > > >>>>>>>>> verification time to
> > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > > initialization
> > > > > > > >>>>>>>>> times
> > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> > wgrib2
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>> MET commands
> > > > > > > >>>>>>>>> to illustrate.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
> forecast
> > > > > > > >>>>>>>>> time
> > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > > sample
> > > > > > > >>>>>>>>> data file you
> > > > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> > > wgrib2
> > > > > > > >>>>>>>>> so
> > > > > > > >>>>>>>>> I have that
> > > > > > > >>>>>>>>> funky forecast hour:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour
ave
> > > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> fcst:vt=2018070605
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
> verbosity
> > > > > > > >>>>>>>>> level
> > > > > > > >>>>>>>>> (-v
> > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data
> it
> > is
> > > > > > > >>>>>>>>> reading:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> # Read the first record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> # Read the second record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Hope that helps.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks,
> > > > > > > >>>>>>>>> John
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran
- NOAA
> > > > > > > >>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Hi, John,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> When I implemented all the changes (and also
set it
> up
> > > so
> > > > > > > >>>>>>>>>> it's
> > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > > >>>>>>>>>>
> > > > >
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > > >>>>>>>>>> -v 4
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> But then I got this error:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array()
- No
> > > > > > > >>>>>>>>>> matching
> > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > > matching
> > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> ERROR :
> > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > > > >>>>>>>>>> forecast
> > > > > > > >>>>>>>>>> data
> > > > > > > >>>>>>>>>> found! Exiting...
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output
> for
> > > the
> > > > > > > >>>>>>>>>> file
> > > > > > > >>>>>>>>>> in this case reads:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >
> > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour ave
> > > fcst:
> > > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You had
> > > > > > > >>>>>>>>>> suggested
> > > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day, but
> > > maybe
> > > > > > > >>>>>>>>>> it needs to be
> > > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Perry,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm confused. In the previous air quality
vx we
> did,
> > > we
> > > > > > > >>>>>>>>>>> used
> > > > > > > >>>>>>>>>>> observation files named:
> > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> And those files don't exist in the July data
you
> just
> > > > > > > >>>>>>>>>>> sent.
> > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> aqm.t12z.anowpm.pb.tm00
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against
> the
> > > > > > > >>>>>>>>>>> BUFR
> > > > > > > >>>>>>>>>>> or
> > > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Let me just tell you the key points for the
config
> > file
> > > > > > > >>>>>>>>>>> settings...
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass
2
> input
> > > > > > > >>>>>>>>>>> files
> > > > > > > >>>>>>>>>>> -
> > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > > >>>>>>>>>>> pb2nc \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc
\
> > > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
> time_summary
> > > > > > > >>>>>>>>>>> like
> > > > > > > >>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0;
} //
> > > > > > > >>>>>>>>>>> previous
> > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every hour
> > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> > when
> > > at
> > > > > > > >>>>>>>>>>> least
> > > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> > station
> > > > > > > >>>>>>>>>>> over the
> > > > > > > >>>>>>>>>>> defined time period. They're identified in
the
> > output
> > > by
> > > > > > > >>>>>>>>>>> the message type:
> > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst" and
> > > "obs"
> > > > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > > > >>>>>>>>>>> Instead
> > > > > > > >>>>>>>>>>> of specifying
> > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
> > > convenient
> > > > > > > >>>>>>>>>>> to specify the
> > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify the
> > > config
> > > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > > >>>>>>>>>>> using environment variables:
> > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>> obs = {
> > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway
> <
> > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
> NOAA
> > > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the
bc grids
> > and
> > > > > > > >>>>>>>>>>>>> focus on
> > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy
> files
> > > > > > > >>>>>>>>>>>>> over
> > > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > > >>>>>>>>>>>>> let him work on the files he has for now
and if
> he
> > > runs
> > > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least
the
> 24-hr
> > > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not
> > for
> > > > > > > >>>>>>>>>>>>>> bias
> > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
> > safe,
> > > > > > > >>>>>>>>>>>>>> you
> > > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia
> for
> > > John
> > > > > > > >>>>>>>>>>>>>> to do
> > > > > > > >>>>>>>>>>>>>> some testing some production files. Do
you
> think
> > I
> > > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files are
OK for
> > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that
the ops
> > > files
> > > > > > > >>>>>>>>>>>>>> should
> > > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
> > parallels.
> > > > > > > >>>>>>>>>>>>>>> We
> > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max
hour
> > labels
> > > > > > > >>>>>>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
> upgrade.
> > > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
> correct
> > > time
> > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send John
> > > over
> > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > > > >>>>>>>>>>>>>>> versions
> > > > > > > >>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have -
but
> they
> > > can
> > > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new does
> > over
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want
> to
> > > use
> > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> > Jianping
> > > > > > > >>>>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
> > management
> > > > > > > >>>>>>>>>>>>>>>> did
> > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with
> the
> > > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation.
> I
> > > > > > > >>>>>>>>>>>>>>>> think
> > > > > > > >>>>>>>>>>>>>>>> I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some
> time
> > > ago,
> > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff. I
> > > think
> > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
> entirely
> > > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and
how to
> > > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> > John
> > > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > > Gotway <
> > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away to
> > > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for
1- and
> > 8-
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> \
> > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>
> > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > > \
> > > > > > > >>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW
\
> > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1,
> > for
> > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
> region
> > > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents
> of
> > > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > > >>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> > > level:17-40
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
> > testing
> > > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> > hours
> > > 17
> > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> > better.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are
in the
> > same
> > > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell
it which
> > one
> > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > > lead_time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps
> are
> > > off
> > > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41
hour
> lead
> > > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-
hour max
> > > value
> > > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid
at 05Z
> > > while
> > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
Point-Stat
> > > config
> > > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and
was
> > fixed.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> > > but I
> > > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look
on
> dakota
> > > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute
> > 24-
> > > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped the
> > > output
> > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands to
> > count
> > > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> > that
> > > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> > 6,555
> > > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations are
> > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that
> file
> > is
> > > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
> grep
> > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated in
> > the
> > > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
> > > duplicates
> > > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason
> why
> > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> > across 2
> > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which
we
> could
> > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard Soh
> <
> > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > > (550.319
> > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check
> in
> > > the
> > > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
> > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
> content
> > > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
> inputs.
> > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files and
> > > will
> > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> > > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change
> > the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
> > currently
> > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
> Soh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to
the
> logic
> > > for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > > 24-hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > > counts:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
> currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript to
> > dump
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> > string
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is
in PB2NC
> > not
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries...
or in
> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> > > Soh <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> > and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the
obs
> > data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28
PM John
> > > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could
you
> > please
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> PB2NC.
> > > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request
> > 24-hourly
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
> end =
> > > 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
> but
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what
> > you'd
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > > "AIRNOW_MAX_240000"
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> caused
> > by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need
2 of
> > them
> > > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44
PM John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> > PrepBUFR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute a
> 24
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
> week.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40
PM
> hsoh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
> > reasonable
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
> data
> > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index
of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file was
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
> without
> > > any
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
> > headers
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> > currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value
from 0
> > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type
for
> > > summary)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable.
> > The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger than
> > > 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be
OK.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> > > Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look
on
> dakota
> > > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump
it to
> > > ascii:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > > /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> pb2nc_output.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists
> all
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But some
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced
> by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > > pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing
or
> table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Wed Jun 05 15:06:28 2019
Hi, John,
Well, it does appear that I got the same error with MET 8.1 as I did
with
the previous version:
ERROR :
ERROR : StringArray::operator[](int) const -> range check error!
ERROR :
Perry
On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry,
>
> Did you ever get this issue resolved from 1 month ago? Or should I
log on
> to WCOSS and take a look?
>
> Thanks,
> John
>
> On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> > Hi, John,
> >
> > Well, I am running the script in batch so I don't control what
node it
> > gets. I can tell you that the working directory is here:
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > And the output log file is here:
> > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
> should
> > help you figure out how the thing is run.
> >
> > Thanks!
> >
> > Perry
> >
> > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Perry,
> > >
> > > We can log on to wcoss and test there instead. Yes, pb2nc does
seem to
> > be
> > > the culprit. Can you point us the wcoss node you're using and
the
> > working
> > > directory there? We basically need to recreate your call to
pb2nc to
> try
> > > to replicate the behavior.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > Hi, David, John,
> > > >
> > > > I am not running on Theia. In fact I think it would take
quite a bit
> > of
> > > > effort to transition this thing to Theia, and I'll need to be
running
> > > this
> > > > verification on prod data anyway, which is only on WCOSS.
> > > >
> > > > However, if you wish to troubleshoot on Theia, you should do
so, and
> I
> > > > think you have all you need to be able to do so, right?
> > > >
> > > > It would seem that the error on my side is in PB2NC, so I'll
probably
> > > need
> > > > to make sure I'm running what you're running.
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Perry -
> > > > > Are you running on theia?
> > > > > John suggests that we try to troubleshoot there ...
> > > > > thanks,
> > > > > David
> > > > >
> > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > > Hi Perry -
> > > > > >
> > > > > > An update on where we are on this issue:
> > > > > > Using the prepda.2019042312.nc file you provided we indeed
> > reproduce
> > > > > > the point_stat error.
> > > > > > The point_stat error is due to only a single entry the
valid time
> > > > > > table,
> > > > > > e.g. from ncdump -v hdr_vld_table
> > > > > > hdr_vld_table =
> > > > > > "20190423_050000",
> > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33, 33,
> ...
> > > > > > which are all out of range except the first.
> > > > > >
> > > > > > If I run pb2nc here with your PREPBUFR data and your
> > PB2NCConfig_aqm
> > > > > > config file, however,
> > > > > > I generate a different prepda.2019042312.nc file that runs
> through
> > > to
> > > > > > completion with point_stat.
> > > > > > Its valid time table:
> > > > > >
> > > > > > hdr_vld_table =
> > > > > > "20190423_030000",
> > > > > > "20190423_010000",
> > > > > > "20190423_020000",
> > > > > > "20190423_230000",
> > > > > > "20190423_150000",
> > > > > > "20190423_090000",
> > > > > > "20190423_070000",
> > > > > > "20190423_060000",
> > > > > > "20190423_210000",
> > > > > > "20190423_050000",
> > > > > > "20190423_170000",
> > > > > > "20190423_100000",
> > > > > > "20190423_200000",
> > > > > > "20190423_140000",
> > > > > > "20190423_130000",
> > > > > > "20190423_160000",
> > > > > > "20190423_040000",
> > > > > > "20190423_180000",
> > > > > > "20190423_080000",
> > > > > > "20190423_120000",
> > > > > > "20190423_110000",
> > > > > > "20190423_220000",
> > > > > > "20190423_190000",
> > > > > > "20190424_000000" ;
> > > > > > }
> > > > > >
> > > > > > So it seems pb2nc is running correctly here, but not on
your
> side.
> > > > > > Let me discuss with John the best way to proceed ...
> > > > > > thanks,
> > > > > > David
> > > > > >
> > > > > >
> > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > > > Hi, everyone,
> > > > > > >
> > > > > > > I am attaching this to this long thread regarding the
daily max
> > of
> > > > > > > ozone
> > > > > > > verification, so we remember what we were doing back
then.
> > > > > > >
> > > > > > > I am now running this again using MET 8.0, and something
seems
> to
> > > not
> > > > > > > be
> > > > > > > working when I run point_stat. I get this error when
running a
> > > daily
> > > > > > > max
> > > > > > > of ozone:
> > > > > > >
> > > > > > > ERROR :
> > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> error!
> > > > > > > ERROR :
> > > > > > >
> > > > > > > I am using my own scripts to run this using MET, so this
is not
> > > using
> > > > > > > METplus.
> > > > > > >
> > > > > > > You can find a log file of this here:
> > > > > > >
> > > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > > It
> > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc code
is
> > working
> > > > > > > fine, I
> > > > > > > believe.
> > > > > > >
> > > > > > > The script I am using to run this is here:
> > > > > > >
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > > >
> > > > > > > The prepbufr files are:
> > > > > > >
> > > > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
and
> > > > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > > >
> > > > > > > The model file is here:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > > >
> > > > > > >
> > > > > > > What could be causing this error?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Perry
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > > johnhg at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Great, glad that did the trick.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
> Affiliate
> > <
> > > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > > >
> > > > > > > >> Well this worked! And Jeff confirmed to me that the
8-hr
> > > average
> > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like
in the
> > 1-hr
> > > > > > > >> avg.
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> Perry
> > > > > > > >>
> > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > > >> <johnhg at ucar.edu>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Perry,
> > > > > > > >>>
> > > > > > > >>> I think the problem is in your setting of
> > > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > > >>>
> > > > > > > >>> Here's your wgrib2 output:
> > > > > > > >>> wgrib2 oz_DAY1
> > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
> ave
> > > > > > > >>> fcst:
> > > > > > > >>>
> > > > > > > >>> The "d=2018081112" tells you the model
initialization time.
> > > The
> > > > > > > >>> valid
> > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
> > valid
> > > 16
> > > > > > > >>> hours
> > > > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> > > that
> > > > > > > >>> wgrib2
> > > > > > > >>> command using the "-vt" argument to list the
verification
> > time.
> > > > > > > >>>
> > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that data
> > > isn't
> > > > > > > >>> valid
> > > > > > > >>> at the time you're requesting.
> > > > > > > >>>
> > > > > > > >>> John
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
> > Affiliate
> > > <
> > > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi, John,
> > > > > > > >>>>
> > > > > > > >>>> I haven't been able to work on this for awhile, but
now
> I'm
> > > > > > > >>>> working on
> > > > > > > >>>> it again but seem to be running into issues.
> > > > > > > >>>>
> > > > > > > >>>> I have a file that has both the daily max for 1-hr
ozone
> and
> > > the
> > > > > > > >>>> daily
> > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > > >>>>
> > > > > > > >>>> 9 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>> wgrib2 oz_DAY1
> > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour ave
> > > fcst:
> > > > > > > >>>> 10 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>>
> > > > > > > >>>> As you can see they have different forecast times.
But
> > when I
> > > > > > > >>>> run
> > > > > > > >>>> point_stat, I get this output:
> > > > > > > >>>>
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>>
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> matching
> > > > > > > >>>> records
> > > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > > > >>>> OZMAX8/L1
> > > > > > > >>>> found in file: oz_DAY1
> > > > > > > >>>> WARNING:
> > > > > > > >>>>
> > > > > > > >>>> Here is the relevant portion of the point_stat
config
> file:
> > > > > > > >>>>
> > > > > > > >>>> fcst = {
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> obs = {
> > > > > > > >>>> convert(x) = x * 10^9;
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> I don't remember if I had this working previously
for both
> > the
> > > > > > > >>>> 1hr
> > > > > > > >>>> and
> > > > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue
> > for
> > > > > > > >>>> the
> > > > > > > >>>> 8hr
> > > > > > > >>>> ozone. Have any thoughts?
> > > > > > > >>>>
> > > > > > > >>>> Thanks!
> > > > > > > >>>>
> > > > > > > >>>> Perry
> > > > > > > >>>>
> > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>> <
> > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> I chose option 2, and that seems to work - I have
a .stat
> > > file
> > > > > > > >>>>> with
> > > > > > > >>>>> output. Now I'll need to do some work here to see
if the
> > > > > > > >>>>> output
> > > > > > > >>>>> looks OK,
> > > > > > > >>>>> and to configure the verification so it verifies
OZMAX1
> and
> > > > > > > >>>>> OZMAX8.
> > > > > > > >>>>>
> > > > > > > >>>>> I also wonder if this would be easily modified for
PM.
> > > > > > > >>>>>
> > > > > > > >>>>> Perry
> > > > > > > >>>>>
> > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> > > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Perry,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
> summaries
> > > > > > > >>>>>> that
> > > > > > > >>>>>> are
> > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
> that
> > > are
> > > > > > > >>>>>> purportedly
> > > > > > > >>>>>> valid at 05Z. And that's the reason why you have
so
> many
> > > obs
> > > > > > > >>>>>> rejected due
> > > > > > > >>>>>> to the valid time.
> > > > > > > >>>>>>
> > > > > > > >>>>>> It's up to you guys to decide what the "correct"
way of
> > > > > > > >>>>>> handling
> > > > > > > >>>>>> this
> > > > > > > >>>>>> is... but here are a couple of options:
> > > > > > > >>>>>>
> > > > > > > >>>>>> (1) You could use the existing setup and address
this
> > logic
> > > in
> > > > > > > >>>>>> Point-Stat config file. Change the matching time
> > obs_window
> > > > > > > >>>>>> from 0 to:
> > > > > > > >>>>>>
> > > > > > > >>>>>> obs_window = {
> > > > > > > >>>>>> beg = 3600;
> > > > > > > >>>>>> end = 3600;
> > > > > > > >>>>>> }
> > > > > > > >>>>>> This tells Point-Stat to match observations which
are
> > > exactly
> > > > > > > >>>>>> 1-
> > > > > > > >>>>>> hour
> > > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > > >>>>>>
> > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary from
> > 05Z
> > > > > > > >>>>>> to
> > > > > > > >>>>>> 05Z
> > > > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting
> > at
> > > 0:
> > > > > > > >>>>>>
> > > > > > > >>>>>> time_summary = {
> > > > > > > >>>>>> flag = TRUE;
> > > > > > > >>>>>> beg = "050000";
> > > > > > > >>>>>> end = "050000";
> > > > > > > >>>>>> ...
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks,
> > > > > > > >>>>>> John
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran -
NOAA
> > > > > > > >>>>>> Affiliate
> > > > > > > >>>>>> <
> > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file,
> > but
> > > > > > > >>>>>>> it's
> > > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 4:
> > > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > observation
> > > > > > > >>>>>>> type
> > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> > > method
> > > > > > > >>>>>>> NEAREST(1),
> > > > > > > >>>>>>> using 0 pairs.
> > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thanks!
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Perry
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > > > >>>>>>> Affiliate
> > > > > > > >>>>>>> <
> > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Hi, John,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Looks like the changing of the time stamp from
_04 to
> > _05
> > > > > > > >>>>>>>> did
> > > > > > > >>>>>>>> the
> > > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> > change.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks!
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Perry
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Perry,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
> the
> > > > > > > >>>>>>>>> timestamp
> > > > > > > >>>>>>>>> of the data from that file. I think switching
the
> > > > > > > >>>>>>>>> verification time to
> > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > > initialization
> > > > > > > >>>>>>>>> times
> > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> > wgrib2
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>> MET commands
> > > > > > > >>>>>>>>> to illustrate.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
> forecast
> > > > > > > >>>>>>>>> time
> > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > > sample
> > > > > > > >>>>>>>>> data file you
> > > > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> > > wgrib2
> > > > > > > >>>>>>>>> so
> > > > > > > >>>>>>>>> I have that
> > > > > > > >>>>>>>>> funky forecast hour:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour
ave
> > > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> fcst:vt=2018070605
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
> verbosity
> > > > > > > >>>>>>>>> level
> > > > > > > >>>>>>>>> (-v
> > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data
> it
> > is
> > > > > > > >>>>>>>>> reading:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> # Read the first record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> # Read the second record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Hope that helps.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks,
> > > > > > > >>>>>>>>> John
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran
- NOAA
> > > > > > > >>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Hi, John,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> When I implemented all the changes (and also
set it
> up
> > > so
> > > > > > > >>>>>>>>>> it's
> > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > > >>>>>>>>>>
> > > > >
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > > >>>>>>>>>> -v 4
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> But then I got this error:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array()
- No
> > > > > > > >>>>>>>>>> matching
> > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > > matching
> > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> ERROR :
> > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > > > >>>>>>>>>> forecast
> > > > > > > >>>>>>>>>> data
> > > > > > > >>>>>>>>>> found! Exiting...
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output
> for
> > > the
> > > > > > > >>>>>>>>>> file
> > > > > > > >>>>>>>>>> in this case reads:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >
> > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour ave
> > > fcst:
> > > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You had
> > > > > > > >>>>>>>>>> suggested
> > > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day, but
> > > maybe
> > > > > > > >>>>>>>>>> it needs to be
> > > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Perry,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm confused. In the previous air quality
vx we
> did,
> > > we
> > > > > > > >>>>>>>>>>> used
> > > > > > > >>>>>>>>>>> observation files named:
> > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> And those files don't exist in the July data
you
> just
> > > > > > > >>>>>>>>>>> sent.
> > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> aqm.t12z.anowpm.pb.tm00
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against
> the
> > > > > > > >>>>>>>>>>> BUFR
> > > > > > > >>>>>>>>>>> or
> > > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Let me just tell you the key points for the
config
> > file
> > > > > > > >>>>>>>>>>> settings...
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass
2
> input
> > > > > > > >>>>>>>>>>> files
> > > > > > > >>>>>>>>>>> -
> > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > > >>>>>>>>>>> pb2nc \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc
\
> > > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
> time_summary
> > > > > > > >>>>>>>>>>> like
> > > > > > > >>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0;
} //
> > > > > > > >>>>>>>>>>> previous
> > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every hour
> > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> > when
> > > at
> > > > > > > >>>>>>>>>>> least
> > > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> > station
> > > > > > > >>>>>>>>>>> over the
> > > > > > > >>>>>>>>>>> defined time period. They're identified in
the
> > output
> > > by
> > > > > > > >>>>>>>>>>> the message type:
> > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst" and
> > > "obs"
> > > > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > > > >>>>>>>>>>> Instead
> > > > > > > >>>>>>>>>>> of specifying
> > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
> > > convenient
> > > > > > > >>>>>>>>>>> to specify the
> > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify the
> > > config
> > > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > > >>>>>>>>>>> using environment variables:
> > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>> obs = {
> > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway
> <
> > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
> NOAA
> > > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the
bc grids
> > and
> > > > > > > >>>>>>>>>>>>> focus on
> > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy
> files
> > > > > > > >>>>>>>>>>>>> over
> > > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > > >>>>>>>>>>>>> let him work on the files he has for now
and if
> he
> > > runs
> > > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least
the
> 24-hr
> > > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not
> > for
> > > > > > > >>>>>>>>>>>>>> bias
> > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
> > safe,
> > > > > > > >>>>>>>>>>>>>> you
> > > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia
> for
> > > John
> > > > > > > >>>>>>>>>>>>>> to do
> > > > > > > >>>>>>>>>>>>>> some testing some production files. Do
you
> think
> > I
> > > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files are
OK for
> > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that
the ops
> > > files
> > > > > > > >>>>>>>>>>>>>> should
> > > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
> > parallels.
> > > > > > > >>>>>>>>>>>>>>> We
> > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max
hour
> > labels
> > > > > > > >>>>>>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
> upgrade.
> > > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
> correct
> > > time
> > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send John
> > > over
> > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > > > >>>>>>>>>>>>>>> versions
> > > > > > > >>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have -
but
> they
> > > can
> > > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new does
> > over
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want
> to
> > > use
> > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> > Jianping
> > > > > > > >>>>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
> > management
> > > > > > > >>>>>>>>>>>>>>>> did
> > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with
> the
> > > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation.
> I
> > > > > > > >>>>>>>>>>>>>>>> think
> > > > > > > >>>>>>>>>>>>>>>> I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some
> time
> > > ago,
> > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff. I
> > > think
> > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
> entirely
> > > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and
how to
> > > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> > John
> > > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > > Gotway <
> > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away to
> > > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for
1- and
> > 8-
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> \
> > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>
> > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > > \
> > > > > > > >>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW
\
> > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1,
> > for
> > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
> region
> > > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents
> of
> > > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > > >>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> > > level:17-40
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
> > testing
> > > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> > hours
> > > 17
> > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> > better.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are
in the
> > same
> > > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell
it which
> > one
> > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > > lead_time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps
> are
> > > off
> > > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41
hour
> lead
> > > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-
hour max
> > > value
> > > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid
at 05Z
> > > while
> > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
Point-Stat
> > > config
> > > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and
was
> > fixed.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> > > but I
> > > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look
on
> dakota
> > > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute
> > 24-
> > > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped the
> > > output
> > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands to
> > count
> > > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> > that
> > > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> > 6,555
> > > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations are
> > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that
> file
> > is
> > > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
> grep
> > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated in
> > the
> > > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
> > > duplicates
> > > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason
> why
> > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> > across 2
> > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which
we
> could
> > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard Soh
> <
> > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > > (550.319
> > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check
> in
> > > the
> > > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
> > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
> content
> > > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
> inputs.
> > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files and
> > > will
> > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> > > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change
> > the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
> > currently
> > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
> Soh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to
the
> logic
> > > for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > > 24-hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > > counts:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
> currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript to
> > dump
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> > string
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is
in PB2NC
> > not
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries...
or in
> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> > > Soh <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> > and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the
obs
> > data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28
PM John
> > > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could
you
> > please
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> PB2NC.
> > > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request
> > 24-hourly
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
> end =
> > > 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
> but
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what
> > you'd
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > > "AIRNOW_MAX_240000"
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> caused
> > by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need
2 of
> > them
> > > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44
PM John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> > PrepBUFR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute a
> 24
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
> week.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40
PM
> hsoh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
> > reasonable
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
> data
> > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index
of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file was
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
> without
> > > any
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
> > headers
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> > currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value
from 0
> > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type
for
> > > summary)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable.
> > The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger than
> > > 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be
OK.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> > > Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look
on
> dakota
> > > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump
it to
> > > ascii:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > > /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> pb2nc_output.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists
> all
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But some
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced
> by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > > pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing
or
> table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Thu Jun 06 15:21:44 2019
Perry,
I was able to replicate this behavior both on WCOSS and locally. I
believe
it's a bug in PB2NC. I'm trying to narrow it down and will then hand
it
off to Howard to debug. Thanks for identifying this issue.
John
On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, John,
>
> Well, it does appear that I got the same error with MET 8.1 as I did
with
> the previous version:
>
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> Perry
>
> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Perry,
> >
> > Did you ever get this issue resolved from 1 month ago? Or should
I log
> on
> > to WCOSS and take a look?
> >
> > Thanks,
> > John
> >
> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Hi, John,
> > >
> > > Well, I am running the script in batch so I don't control what
node it
> > > gets. I can tell you that the working directory is here:
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >
> > > And the output log file is here:
> > > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
> > should
> > > help you figure out how the thing is run.
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Perry,
> > > >
> > > > We can log on to wcoss and test there instead. Yes, pb2nc
does seem
> to
> > > be
> > > > the culprit. Can you point us the wcoss node you're using and
the
> > > working
> > > > directory there? We basically need to recreate your call to
pb2nc to
> > try
> > > > to replicate the behavior.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > > > >
> > > > > Hi, David, John,
> > > > >
> > > > > I am not running on Theia. In fact I think it would take
quite a
> bit
> > > of
> > > > > effort to transition this thing to Theia, and I'll need to
be
> running
> > > > this
> > > > > verification on prod data anyway, which is only on WCOSS.
> > > > >
> > > > > However, if you wish to troubleshoot on Theia, you should do
so,
> and
> > I
> > > > > think you have all you need to be able to do so, right?
> > > > >
> > > > > It would seem that the error on my side is in PB2NC, so I'll
> probably
> > > > need
> > > > > to make sure I'm running what you're running.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Perry
> > > > >
> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Perry -
> > > > > > Are you running on theia?
> > > > > > John suggests that we try to troubleshoot there ...
> > > > > > thanks,
> > > > > > David
> > > > > >
> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > > > Hi Perry -
> > > > > > >
> > > > > > > An update on where we are on this issue:
> > > > > > > Using the prepda.2019042312.nc file you provided we
indeed
> > > reproduce
> > > > > > > the point_stat error.
> > > > > > > The point_stat error is due to only a single entry the
valid
> time
> > > > > > > table,
> > > > > > > e.g. from ncdump -v hdr_vld_table
> > > > > > > hdr_vld_table =
> > > > > > > "20190423_050000",
> > > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33, 33,
> > ...
> > > > > > > which are all out of range except the first.
> > > > > > >
> > > > > > > If I run pb2nc here with your PREPBUFR data and your
> > > PB2NCConfig_aqm
> > > > > > > config file, however,
> > > > > > > I generate a different prepda.2019042312.nc file that
runs
> > through
> > > > to
> > > > > > > completion with point_stat.
> > > > > > > Its valid time table:
> > > > > > >
> > > > > > > hdr_vld_table =
> > > > > > > "20190423_030000",
> > > > > > > "20190423_010000",
> > > > > > > "20190423_020000",
> > > > > > > "20190423_230000",
> > > > > > > "20190423_150000",
> > > > > > > "20190423_090000",
> > > > > > > "20190423_070000",
> > > > > > > "20190423_060000",
> > > > > > > "20190423_210000",
> > > > > > > "20190423_050000",
> > > > > > > "20190423_170000",
> > > > > > > "20190423_100000",
> > > > > > > "20190423_200000",
> > > > > > > "20190423_140000",
> > > > > > > "20190423_130000",
> > > > > > > "20190423_160000",
> > > > > > > "20190423_040000",
> > > > > > > "20190423_180000",
> > > > > > > "20190423_080000",
> > > > > > > "20190423_120000",
> > > > > > > "20190423_110000",
> > > > > > > "20190423_220000",
> > > > > > > "20190423_190000",
> > > > > > > "20190424_000000" ;
> > > > > > > }
> > > > > > >
> > > > > > > So it seems pb2nc is running correctly here, but not on
your
> > side.
> > > > > > > Let me discuss with John the best way to proceed ...
> > > > > > > thanks,
> > > > > > > David
> > > > > > >
> > > > > > >
> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
wrote:
> > > > > > > > Hi, everyone,
> > > > > > > >
> > > > > > > > I am attaching this to this long thread regarding the
daily
> max
> > > of
> > > > > > > > ozone
> > > > > > > > verification, so we remember what we were doing back
then.
> > > > > > > >
> > > > > > > > I am now running this again using MET 8.0, and
something
> seems
> > to
> > > > not
> > > > > > > > be
> > > > > > > > working when I run point_stat. I get this error when
> running a
> > > > daily
> > > > > > > > max
> > > > > > > > of ozone:
> > > > > > > >
> > > > > > > > ERROR :
> > > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> > error!
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > I am using my own scripts to run this using MET, so
this is
> not
> > > > using
> > > > > > > > METplus.
> > > > > > > >
> > > > > > > > You can find a log file of this here:
> > > > > > > >
> > > > >
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > > > It
> > > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc
code is
> > > working
> > > > > > > > fine, I
> > > > > > > > believe.
> > > > > > > >
> > > > > > > > The script I am using to run this is here:
> > > > > > > >
> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > > > >
> > > > > > > > The prepbufr files are:
> > > > > > > >
> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > > > >
> > > > > > > > The model file is here:
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > > > >
> > > > > > > >
> > > > > > > > What could be causing this error?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Perry
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > > > johnhg at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Great, glad that did the trick.
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
> > Affiliate
> > > <
> > > > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > >> Well this worked! And Jeff confirmed to me that
the 8-hr
> > > > average
> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like
in the
> > > 1-hr
> > > > > > > > >> avg.
> > > > > > > > >>
> > > > > > > > >> Thanks!
> > > > > > > > >>
> > > > > > > > >> Perry
> > > > > > > > >>
> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > > > >> <johnhg at ucar.edu>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> Perry,
> > > > > > > > >>>
> > > > > > > > >>> I think the problem is in your setting of
> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > > > >>>
> > > > > > > > >>> Here's your wgrib2 output:
> > > > > > > > >>> wgrib2 oz_DAY1
> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour ave
> > fcst:
> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-
22 hour
> > ave
> > > > > > > > >>> fcst:
> > > > > > > > >>>
> > > > > > > > >>> The "d=2018081112" tells you the model
initialization
> time.
> > > > The
> > > > > > > > >>> valid
> > > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
> > > valid
> > > > 16
> > > > > > > > >>> hours
> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
later.
> Rerun
> > > > that
> > > > > > > > >>> wgrib2
> > > > > > > > >>> command using the "-vt" argument to list the
verification
> > > time.
> > > > > > > > >>>
> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that
> data
> > > > isn't
> > > > > > > > >>> valid
> > > > > > > > >>> at the time you're requesting.
> > > > > > > > >>>
> > > > > > > > >>> John
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran -
NOAA
> > > Affiliate
> > > > <
> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi, John,
> > > > > > > > >>>>
> > > > > > > > >>>> I haven't been able to work on this for awhile,
but now
> > I'm
> > > > > > > > >>>> working on
> > > > > > > > >>>> it again but seem to be running into issues.
> > > > > > > > >>>>
> > > > > > > > >>>> I have a file that has both the daily max for 1-
hr ozone
> > and
> > > > the
> > > > > > > > >>>> daily
> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > > > >>>>
> > > > > > > > >>>> 9 (t10a2)
> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > > >>>> >
> > > > > > > > >>>> wgrib2 oz_DAY1
> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
> ave
> > > > fcst:
> > > > > > > > >>>> 10 (t10a2)
> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > > >>>> >
> > > > > > > > >>>>
> > > > > > > > >>>> As you can see they have different forecast
times. But
> > > when I
> > > > > > > > >>>> run
> > > > > > > > >>>> point_stat, I get this output:
> > > > > > > > >>>>
> > > > > > > > >>>> DEBUG 2:
> > > > > > > > >>>> DEBUG 2:
> > > > > > > > >>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > > >>>> DEBUG 2:
> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
> > matching
> > > > > > > > >>>> records
> > > > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
> matching
> > > > > > > > >>>> OZMAX8/L1
> > > > > > > > >>>> found in file: oz_DAY1
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>>
> > > > > > > > >>>> Here is the relevant portion of the point_stat
config
> > file:
> > > > > > > > >>>>
> > > > > > > > >>>> fcst = {
> > > > > > > > >>>> field = [
> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > > >>>> ];
> > > > > > > > >>>> }
> > > > > > > > >>>>
> > > > > > > > >>>> obs = {
> > > > > > > > >>>> convert(x) = x * 10^9;
> > > > > > > > >>>> field = [
> > > > > > > > >>>> { name = "COPO"; level = "A1"; message_type
=
> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > > > >>>> { name = "COPO"; level = "A8"; message_type
=
> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > > > >>>> ];
> > > > > > > > >>>> }
> > > > > > > > >>>>
> > > > > > > > >>>> I don't remember if I had this working previously
for
> both
> > > the
> > > > > > > > >>>> 1hr
> > > > > > > > >>>> and
> > > > > > > > >>>> the 8hr ozone, but apparently there is some sort
of
> issue
> > > for
> > > > > > > > >>>> the
> > > > > > > > >>>> 8hr
> > > > > > > > >>>> ozone. Have any thoughts?
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks!
> > > > > > > > >>>>
> > > > > > > > >>>> Perry
> > > > > > > > >>>>
> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
> > > > Affiliate
> > > > > > > > >>>> <
> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> I chose option 2, and that seems to work - I
have a
> .stat
> > > > file
> > > > > > > > >>>>> with
> > > > > > > > >>>>> output. Now I'll need to do some work here to
see if
> the
> > > > > > > > >>>>> output
> > > > > > > > >>>>> looks OK,
> > > > > > > > >>>>> and to configure the verification so it verifies
OZMAX1
> > and
> > > > > > > > >>>>> OZMAX8.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I also wonder if this would be easily modified
for PM.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Perry
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> > > > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Perry,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute
time
> > summaries
> > > > > > > > >>>>>> that
> > > > > > > > >>>>>> are
> > > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
> > that
> > > > are
> > > > > > > > >>>>>> purportedly
> > > > > > > > >>>>>> valid at 05Z. And that's the reason why you
have so
> > many
> > > > obs
> > > > > > > > >>>>>> rejected due
> > > > > > > > >>>>>> to the valid time.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> It's up to you guys to decide what the
"correct" way
> of
> > > > > > > > >>>>>> handling
> > > > > > > > >>>>>> this
> > > > > > > > >>>>>> is... but here are a couple of options:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> (1) You could use the existing setup and
address this
> > > logic
> > > > in
> > > > > > > > >>>>>> Point-Stat config file. Change the matching
time
> > > obs_window
> > > > > > > > >>>>>> from 0 to:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> obs_window = {
> > > > > > > > >>>>>> beg = 3600;
> > > > > > > > >>>>>> end = 3600;
> > > > > > > > >>>>>> }
> > > > > > > > >>>>>> This tells Point-Stat to match observations
which are
> > > > exactly
> > > > > > > > >>>>>> 1-
> > > > > > > > >>>>>> hour
> > > > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary
> from
> > > 05Z
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>> 05Z
> > > > > > > > >>>>>> instead and then leave the Point-Stat
obs_window
> setting
> > > at
> > > > 0:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> time_summary = {
> > > > > > > > >>>>>> flag = TRUE;
> > > > > > > > >>>>>> beg = "050000";
> > > > > > > > >>>>>> end = "050000";
> > > > > > > > >>>>>> ...
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks,
> > > > > > > > >>>>>> John
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran
- NOAA
> > > > > > > > >>>>>> Affiliate
> > > > > > > > >>>>>> <
> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
output
> file,
> > > but
> > > > > > > > >>>>>>> it's
> > > > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> DEBUG 4:
> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > > observation
> > > > > > > > >>>>>>> type
> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
> interpolation
> > > > method
> > > > > > > > >>>>>>> NEAREST(1),
> > > > > > > > >>>>>>> using 0 pairs.
> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thanks!
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Perry
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran
- NOAA
> > > > > > > > >>>>>>> Affiliate
> > > > > > > > >>>>>>> <
> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Hi, John,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Looks like the changing of the time stamp
from _04
> to
> > > _05
> > > > > > > > >>>>>>>> did
> > > > > > > > >>>>>>>> the
> > > > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> > > change.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thanks!
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Perry
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
> Gotway <
> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Perry,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
> > the
> > > > > > > > >>>>>>>>> timestamp
> > > > > > > > >>>>>>>>> of the data from that file. I think
switching the
> > > > > > > > >>>>>>>>> verification time to
> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix
it.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > > > initialization
> > > > > > > > >>>>>>>>> times
> > > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> > > wgrib2
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>> MET commands
> > > > > > > > >>>>>>>>> to illustrate.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-
t),
> > forecast
> > > > > > > > >>>>>>>>> time
> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > > > sample
> > > > > > > > >>>>>>>>> data file you
> > > > > > > > >>>>>>>>> sent to me. But remember I'm still using
the "old"
> > > > wgrib2
> > > > > > > > >>>>>>>>> so
> > > > > > > > >>>>>>>>> I have that
> > > > > > > > >>>>>>>>> funky forecast hour:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624
hour ave
> > > > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> > fcst:vt=2018070605
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
> > verbosity
> > > > > > > > >>>>>>>>> level
> > > > > > > > >>>>>>>>> (-v
> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of
the data
> > it
> > > is
> > > > > > > > >>>>>>>>> reading:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> # Read the first record
> > > > > > > > >>>>>>>>> plot_data_plane
> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > > > >>>>>>>>> ...
> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > > >>>>>>>>> ...
> > > > > > > > >>>>>>>>> # Read the second record
> > > > > > > > >>>>>>>>> plot_data_plane
> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > > > >>>>>>>>> ...
> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Hope that helps.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>> John
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
Shafran -
> NOAA
> > > > > > > > >>>>>>>>> Affiliate <
> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Hi, John,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> When I implemented all the changes (and
also set
> it
> > up
> > > > so
> > > > > > > > >>>>>>>>>> it's
> > > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > > > >>>>>>>>>>
> > > > > >
> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > > > >>>>>>>>>> -v 4
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> But then I got this error:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> WARNING:
MetGrib2DataFile::data_plane_array() - No
> > > > > > > > >>>>>>>>>> matching
> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > > > matching
> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> ERROR :
> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
> requested
> > > > > > > > >>>>>>>>>> forecast
> > > > > > > > >>>>>>>>>> data
> > > > > > > > >>>>>>>>>> found! Exiting...
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output
> > for
> > > > the
> > > > > > > > >>>>>>>>>> file
> > > > > > > > >>>>>>>>>> in this case reads:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > > > >>>>>>>>>>
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >
> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour
> ave
> > > > fcst:
> > > > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > > > >>>>>>>>>>
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You
> had
> > > > > > > > >>>>>>>>>> suggested
> > > > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day,
> but
> > > > maybe
> > > > > > > > >>>>>>>>>> it needs to be
> > > > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
> elsewhere.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
> Gotway <
> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Perry,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air quality
vx we
> > did,
> > > > we
> > > > > > > > >>>>>>>>>>> used
> > > > > > > > >>>>>>>>>>> observation files named:
> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs
and
> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> And those files don't exist in the July
data you
> > just
> > > > > > > > >>>>>>>>>>> sent.
> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> > aqm.t12z.anowpm.pb.tm00
> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against
> > the
> > > > > > > > >>>>>>>>>>> BUFR
> > > > > > > > >>>>>>>>>>> or
> > > > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Let me just tell you the key points for
the
> config
> > > file
> > > > > > > > >>>>>>>>>>> settings...
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to
pass 2
> > input
> > > > > > > > >>>>>>>>>>> files
> > > > > > > > >>>>>>>>>>> -
> > > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > > > >>>>>>>>>>> pb2nc \
> > > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00
\
> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
> > time_summary
> > > > > > > > >>>>>>>>>>> like
> > > > > > > > >>>>>>>>>>> this:
> > > > > > > > >>>>>>>>>>> time_summary = {
> > > > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > > > >>>>>>>>>>> end = "040000";
> > > > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end =
0; } //
> > > > > > > > >>>>>>>>>>> previous
> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every
> hour
> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> > > when
> > > > at
> > > > > > > > >>>>>>>>>>> least
> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > > > >>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> > > station
> > > > > > > > >>>>>>>>>>> over the
> > > > > > > > >>>>>>>>>>> defined time period. They're identified
in the
> > > output
> > > > by
> > > > > > > > >>>>>>>>>>> the message type:
> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst"
> and
> > > > "obs"
> > > > > > > > >>>>>>>>>>> sections like this for the max of the 1-
hour
> data.
> > > > > > > > >>>>>>>>>>> Instead
> > > > > > > > >>>>>>>>>>> of specifying
> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
> > > > convenient
> > > > > > > > >>>>>>>>>>> to specify the
> > > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify the
> > > > config
> > > > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > > > >>>>>>>>>>> using environment variables:
> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> fcst = {
> > > > > > > > >>>>>>>>>>> field = [
> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > > >>>>>>>>>>> ];
> > > > > > > > >>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>> obs = {
> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > > > >>>>>>>>>>> field = [
> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
> message_type =
> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > > > >>>>>>>>>>> ];
> > > > > > > > >>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
Halley
> Gotway
> > <
> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
> > NOAA
> > > > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the
bc
> grids
> > > and
> > > > > > > > >>>>>>>>>>>>> focus on
> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy
> > files
> > > > > > > > >>>>>>>>>>>>> over
> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > > > >>>>>>>>>>>>> let him work on the files he has for now
and if
> > he
> > > > runs
> > > > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
least the
> > 24-hr
> > > > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
> grids...not
> > > for
> > > > > > > > >>>>>>>>>>>>>> bias
> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
> > > safe,
> > > > > > > > >>>>>>>>>>>>>> you
> > > > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia
> > for
> > > > John
> > > > > > > > >>>>>>>>>>>>>> to do
> > > > > > > > >>>>>>>>>>>>>> some testing some production files. Do
you
> > think
> > > I
> > > > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files
are OK
> for
> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think
that the
> ops
> > > > files
> > > > > > > > >>>>>>>>>>>>>> should
> > > > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about
the
> > > parallels.
> > > > > > > > >>>>>>>>>>>>>>> We
> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily
max hour
> > > labels
> > > > > > > > >>>>>>>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19
CMAQ
> > upgrade.
> > > > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
> > > > Affiliate
> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have
the
> > correct
> > > > time
> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send
> John
> > > > over
> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has
any other
> > > > > > > > >>>>>>>>>>>>>>> versions
> > > > > > > > >>>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have
- but
> > they
> > > > can
> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new
> does
> > > over
> > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
> jeff.mcqueen <
> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you
may want
> > to
> > > > use
> > > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> > > Jianping
> > > > > > > > >>>>>>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it
but
> > > management
> > > > > > > > >>>>>>>>>>>>>>>> did
> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do
so with
> > the
> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
> > > > Affiliate
> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation.
> > I
> > > > > > > > >>>>>>>>>>>>>>>> think
> > > > > > > > >>>>>>>>>>>>>>>> I
> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some
> > time
> > > > ago,
> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff.
> I
> > > > think
> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not
too
> > entirely
> > > > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct
and how
> to
> > > > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> > > John
> > > > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > > > Gotway <
> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away
> to
> > > > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values
for 1-
> and
> > > 8-
> > > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > \
> > > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > > > \
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
PointStatConfig_AIRNOW \
> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
> COPO/A1,
> > > for
> > > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
> > region
> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
> contents
> > of
> > > > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour
ave
> fcst:
> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
sigma
> > > > level:17-40
> > > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
> > > testing
> > > > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> > > hours
> > > > 17
> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> > > better.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours
are in
> the
> > > same
> > > > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell
it
> which
> > > one
> > > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > > > lead_time
> > > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps
> > are
> > > > off
> > > > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a
41 hour
> > lead
> > > > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-
hour
> max
> > > > value
> > > > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
valid at
> 05Z
> > > > while
> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
Point-Stat
> > > > config
> > > > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back
1
> hour:
> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files
and was
> > > fixed.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
> summaries,
> > > > but I
> > > > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look
on
> > dakota
> > > > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc
to
> compute
> > > 24-
> > > > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped
> the
> > > > output
> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands to
> > > count
> > > > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> > > pb2nc_output.txt
> > > > |
> > > > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> > > pb2nc_output.txt
> > > > |
> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> > pb2nc_output.txt |
> > > > wc
> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> > pb2nc_output.txt |
> > > > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> > > that
> > > > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
unique.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> > > 6,555
> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations
> are
> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that
> > file
> > > is
> > > > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
> > grep
> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated
> in
> > > the
> > > > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
> > > > duplicates
> > > > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM
hsoh
> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason
> > why
> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> > > across 2
> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from
which we
> > could
> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard
> Soh
> > <
> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > > > (550.319
> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
minutes.
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will
not check
> > in
> > > > the
> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
files.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
> > content
> > > > of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
> > > summary
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple
BURF
> > inputs.
> > > I
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files
> and
> > > > will
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
Halley
> Gotway
> > > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this
will
> change
> > > the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case
not
> > > currently
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
> > Soh
> > > <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
> > > summary
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to
SVN.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
> /d3/personal/hsoh/data/pb2nc_summary/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> > > Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to
the
> > logic
> > > > for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > > > 24-hour
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > > > counts:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
> > currently)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript
> to
> > > dump
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> > > string
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is
in
> PB2NC
> > > not
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
summaries... or in
> > the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54
PM John
> > > Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41
PM
> Howard
> > > > Soh <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> > > and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for
the obs
> > > data.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28
PM
> John
> > > > Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
Could you
> > > please
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> > PB2NC.
> > > > I
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig
\
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
request
> > > 24-hourly
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z,
like
> this:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
> > end =
> > > > 0;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
> > but
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is
not what
> > > you'd
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > > > "AIRNOW_MAX_240000"
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
referenced.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> > caused
> > > by
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I
need 2 of
> > > them
> > > > to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to
04Z
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a
look?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:44 PM
> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> > > PrepBUFR
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute
> a
> > 24
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
> > week.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:40 PM
> > hsoh
> > > <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script
is
> > > reasonable
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
> > data
> > > in
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
index of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file was
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from
pb2nc
> > without
> > > > any
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
> > > headers
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
> > NetCDF
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(7295
> > > currently)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
value
> from 0
> > > to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message
type for
> > > > summary)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
> variable.
> > > The
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger
> than
> > > > 3655.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to
be OK.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >
> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM,
John
> Halley
> > > > Gotway
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a
look on
> > dakota
> > > > in
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to
dump it
> to
> > > > ascii:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > > > /d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> > pb2nc_output.txt
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists
> > all
> > > of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But
> some
> > > of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
> evidenced
> > by
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > > > pb2nc_output.nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
indexing or
> > table
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Thu Jun 06 16:21:19 2019
Perry (and Howard),
I isolated this issue down and wrote up a GitHub issue describing the
problem:
https://github.com/NCAR/MET/issues/1137
Howard, please take a look and let me know if you need any help
further
debugging this.
Thanks,
John
On Thu, Jun 6, 2019 at 3:21 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Perry,
>
> I was able to replicate this behavior both on WCOSS and locally. I
> believe it's a bug in PB2NC. I'm trying to narrow it down and will
then
> hand it off to Howard to debug. Thanks for identifying this issue.
>
> John
>
> On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>>
>> Hi, John,
>>
>> Well, it does appear that I got the same error with MET 8.1 as I
did with
>> the previous version:
>>
>> ERROR :
>> ERROR : StringArray::operator[](int) const -> range check error!
>> ERROR :
>>
>> Perry
>>
>> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Perry,
>> >
>> > Did you ever get this issue resolved from 1 month ago? Or should
I log
>> on
>> > to WCOSS and take a look?
>> >
>> > Thanks,
>> > John
>> >
>> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Hi, John,
>> > >
>> > > Well, I am running the script in batch so I don't control what
node it
>> > > gets. I can tell you that the working directory is here:
>> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > >
>> > > And the output log file is here:
>> > > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
>> > should
>> > > help you figure out how the thing is run.
>> > >
>> > > Thanks!
>> > >
>> > > Perry
>> > >
>> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
>> > > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Perry,
>> > > >
>> > > > We can log on to wcoss and test there instead. Yes, pb2nc
does
>> seem to
>> > > be
>> > > > the culprit. Can you point us the wcoss node you're using
and the
>> > > working
>> > > > directory there? We basically need to recreate your call to
pb2nc
>> to
>> > try
>> > > > to replicate the behavior.
>> > > >
>> > > > Thanks,
>> > > > John
>> > > >
>> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT
<
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>> > > > >
>> > > > > Hi, David, John,
>> > > > >
>> > > > > I am not running on Theia. In fact I think it would take
quite a
>> bit
>> > > of
>> > > > > effort to transition this thing to Theia, and I'll need to
be
>> running
>> > > > this
>> > > > > verification on prod data anyway, which is only on WCOSS.
>> > > > >
>> > > > > However, if you wish to troubleshoot on Theia, you should
do so,
>> and
>> > I
>> > > > > think you have all you need to be able to do so, right?
>> > > > >
>> > > > > It would seem that the error on my side is in PB2NC, so
I'll
>> probably
>> > > > need
>> > > > > to make sure I'm running what you're running.
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > Perry
>> > > > >
>> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
>> > > met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Perry -
>> > > > > > Are you running on theia?
>> > > > > > John suggests that we try to troubleshoot there ...
>> > > > > > thanks,
>> > > > > > David
>> > > > > >
>> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
>> > > > > > > Hi Perry -
>> > > > > > >
>> > > > > > > An update on where we are on this issue:
>> > > > > > > Using the prepda.2019042312.nc file you provided we
indeed
>> > > reproduce
>> > > > > > > the point_stat error.
>> > > > > > > The point_stat error is due to only a single entry the
valid
>> time
>> > > > > > > table,
>> > > > > > > e.g. from ncdump -v hdr_vld_table
>> > > > > > > hdr_vld_table =
>> > > > > > > "20190423_050000",
>> > > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33,
>> 33,
>> > ...
>> > > > > > > which are all out of range except the first.
>> > > > > > >
>> > > > > > > If I run pb2nc here with your PREPBUFR data and your
>> > > PB2NCConfig_aqm
>> > > > > > > config file, however,
>> > > > > > > I generate a different prepda.2019042312.nc file that
runs
>> > through
>> > > > to
>> > > > > > > completion with point_stat.
>> > > > > > > Its valid time table:
>> > > > > > >
>> > > > > > > hdr_vld_table =
>> > > > > > > "20190423_030000",
>> > > > > > > "20190423_010000",
>> > > > > > > "20190423_020000",
>> > > > > > > "20190423_230000",
>> > > > > > > "20190423_150000",
>> > > > > > > "20190423_090000",
>> > > > > > > "20190423_070000",
>> > > > > > > "20190423_060000",
>> > > > > > > "20190423_210000",
>> > > > > > > "20190423_050000",
>> > > > > > > "20190423_170000",
>> > > > > > > "20190423_100000",
>> > > > > > > "20190423_200000",
>> > > > > > > "20190423_140000",
>> > > > > > > "20190423_130000",
>> > > > > > > "20190423_160000",
>> > > > > > > "20190423_040000",
>> > > > > > > "20190423_180000",
>> > > > > > > "20190423_080000",
>> > > > > > > "20190423_120000",
>> > > > > > > "20190423_110000",
>> > > > > > > "20190423_220000",
>> > > > > > > "20190423_190000",
>> > > > > > > "20190424_000000" ;
>> > > > > > > }
>> > > > > > >
>> > > > > > > So it seems pb2nc is running correctly here, but not on
your
>> > side.
>> > > > > > > Let me discuss with John the best way to proceed ...
>> > > > > > > thanks,
>> > > > > > > David
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
wrote:
>> > > > > > > > Hi, everyone,
>> > > > > > > >
>> > > > > > > > I am attaching this to this long thread regarding the
daily
>> max
>> > > of
>> > > > > > > > ozone
>> > > > > > > > verification, so we remember what we were doing back
then.
>> > > > > > > >
>> > > > > > > > I am now running this again using MET 8.0, and
something
>> seems
>> > to
>> > > > not
>> > > > > > > > be
>> > > > > > > > working when I run point_stat. I get this error when
>> running a
>> > > > daily
>> > > > > > > > max
>> > > > > > > > of ozone:
>> > > > > > > >
>> > > > > > > > ERROR :
>> > > > > > > > ERROR : StringArray::operator[](int) const -> range
check
>> > error!
>> > > > > > > > ERROR :
>> > > > > > > >
>> > > > > > > > I am using my own scripts to run this using MET, so
this is
>> not
>> > > > using
>> > > > > > > > METplus.
>> > > > > > > >
>> > > > > > > > You can find a log file of this here:
>> > > > > > > >
>> > > > >
>> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
>> > > > > > > > It
>> > > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc
code is
>> > > working
>> > > > > > > > fine, I
>> > > > > > > > believe.
>> > > > > > > >
>> > > > > > > > The script I am using to run this is here:
>> > > > > > > >
>> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
>> > > > > > > >
>> > > > > > > > The prepbufr files are:
>> > > > > > > >
>> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
>> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
>> > > > > > > >
>> > > > > > > > The model file is here:
>> > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > What could be causing this error?
>> > > > > > > >
>> > > > > > > > Thanks!
>> > > > > > > >
>> > > > > > > > Perry
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
>> > > > johnhg at ucar.edu>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Great, glad that did the trick.
>> > > > > > > > >
>> > > > > > > > > John
>> > > > > > > > >
>> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran -
NOAA
>> > Affiliate
>> > > <
>> > > > > > > > > perry.shafran at noaa.gov> wrote:
>> > > > > > > > >
>> > > > > > > > >> Well this worked! And Jeff confirmed to me that
the 8-hr
>> > > > average
>> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z
like in
>> the
>> > > 1-hr
>> > > > > > > > >> avg.
>> > > > > > > > >>
>> > > > > > > > >> Thanks!
>> > > > > > > > >>
>> > > > > > > > >> Perry
>> > > > > > > > >>
>> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley
Gotway
>> > > > > > > > >> <johnhg at ucar.edu>
>> > > > > > > > >> wrote:
>> > > > > > > > >>
>> > > > > > > > >>> Perry,
>> > > > > > > > >>>
>> > > > > > > > >>> I think the problem is in your setting of
>> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
>> > > > > > > > >>>
>> > > > > > > > >>> Here's your wgrib2 output:
>> > > > > > > > >>> wgrib2 oz_DAY1
>> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour ave
>> > fcst:
>> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-
22 hour
>> > ave
>> > > > > > > > >>> fcst:
>> > > > > > > > >>>
>> > > > > > > > >>> The "d=2018081112" tells you the model
initialization
>> time.
>> > > > The
>> > > > > > > > >>> valid
>> > > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
>> > > valid
>> > > > 16
>> > > > > > > > >>> hours
>> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
later.
>> Rerun
>> > > > that
>> > > > > > > > >>> wgrib2
>> > > > > > > > >>> command using the "-vt" argument to list the
>> verification
>> > > time.
>> > > > > > > > >>>
>> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that
>> data
>> > > > isn't
>> > > > > > > > >>> valid
>> > > > > > > > >>> at the time you're requesting.
>> > > > > > > > >>>
>> > > > > > > > >>> John
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran -
NOAA
>> > > Affiliate
>> > > > <
>> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>
>> > > > > > > > >>>> Hi, John,
>> > > > > > > > >>>>
>> > > > > > > > >>>> I haven't been able to work on this for awhile,
but now
>> > I'm
>> > > > > > > > >>>> working on
>> > > > > > > > >>>> it again but seem to be running into issues.
>> > > > > > > > >>>>
>> > > > > > > > >>>> I have a file that has both the daily max for 1-
hr
>> ozone
>> > and
>> > > > the
>> > > > > > > > >>>> daily
>> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks
like this:
>> > > > > > > > >>>>
>> > > > > > > > >>>> 9 (t10a2)
>> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > > > > > >>>> >
>> > > > > > > > >>>> wgrib2 oz_DAY1
>> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
>> fcst:
>> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
>> ave
>> > > > fcst:
>> > > > > > > > >>>> 10 (t10a2)
>> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > > > > > >>>> >
>> > > > > > > > >>>>
>> > > > > > > > >>>> As you can see they have different forecast
times. But
>> > > when I
>> > > > > > > > >>>> run
>> > > > > > > > >>>> point_stat, I get this output:
>> > > > > > > > >>>>
>> > > > > > > > >>>> DEBUG 2:
>> > > > > > > > >>>> DEBUG 2:
>> > > > > > > > >>>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------
>> > > > > > > > >>>> DEBUG 2:
>> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
>> > matching
>> > > > > > > > >>>> records
>> > > > > > > > >>>> found for 'OZMAX8/L1'
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
>> matching
>> > > > > > > > >>>> OZMAX8/L1
>> > > > > > > > >>>> found in file: oz_DAY1
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Here is the relevant portion of the point_stat
config
>> > file:
>> > > > > > > > >>>>
>> > > > > > > > >>>> fcst = {
>> > > > > > > > >>>> field = [
>> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
>> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
>> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
>> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
>> > > > > > > > >>>> ];
>> > > > > > > > >>>> }
>> > > > > > > > >>>>
>> > > > > > > > >>>> obs = {
>> > > > > > > > >>>> convert(x) = x * 10^9;
>> > > > > > > > >>>> field = [
>> > > > > > > > >>>> { name = "COPO"; level = "A1";
message_type =
>> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
>> > > > > > > > >>>> { name = "COPO"; level = "A8";
message_type =
>> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
>> > > > > > > > >>>> ];
>> > > > > > > > >>>> }
>> > > > > > > > >>>>
>> > > > > > > > >>>> I don't remember if I had this working
previously for
>> both
>> > > the
>> > > > > > > > >>>> 1hr
>> > > > > > > > >>>> and
>> > > > > > > > >>>> the 8hr ozone, but apparently there is some sort
of
>> issue
>> > > for
>> > > > > > > > >>>> the
>> > > > > > > > >>>> 8hr
>> > > > > > > > >>>> ozone. Have any thoughts?
>> > > > > > > > >>>>
>> > > > > > > > >>>> Thanks!
>> > > > > > > > >>>>
>> > > > > > > > >>>> Perry
>> > > > > > > > >>>>
>> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
>> > > > Affiliate
>> > > > > > > > >>>> <
>> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>
>> > > > > > > > >>>>> I chose option 2, and that seems to work - I
have a
>> .stat
>> > > > file
>> > > > > > > > >>>>> with
>> > > > > > > > >>>>> output. Now I'll need to do some work here to
see if
>> the
>> > > > > > > > >>>>> output
>> > > > > > > > >>>>> looks OK,
>> > > > > > > > >>>>> and to configure the verification so it
verifies
>> OZMAX1
>> > and
>> > > > > > > > >>>>> OZMAX8.
>> > > > > > > > >>>>>
>> > > > > > > > >>>>> I also wonder if this would be easily modified
for PM.
>> > > > > > > > >>>>>
>> > > > > > > > >>>>> Perry
>> > > > > > > > >>>>>
>> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
>> > > > > > > > >>>>> <johnhg at ucar.edu>
>> > > > > > > > >>>>> wrote:
>> > > > > > > > >>>>>
>> > > > > > > > >>>>>> Perry,
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute
time
>> > summaries
>> > > > > > > > >>>>>> that
>> > > > > > > > >>>>>> are
>> > > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
>> > that
>> > > > are
>> > > > > > > > >>>>>> purportedly
>> > > > > > > > >>>>>> valid at 05Z. And that's the reason why you
have so
>> > many
>> > > > obs
>> > > > > > > > >>>>>> rejected due
>> > > > > > > > >>>>>> to the valid time.
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> It's up to you guys to decide what the
"correct" way
>> of
>> > > > > > > > >>>>>> handling
>> > > > > > > > >>>>>> this
>> > > > > > > > >>>>>> is... but here are a couple of options:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> (1) You could use the existing setup and
address this
>> > > logic
>> > > > in
>> > > > > > > > >>>>>> Point-Stat config file. Change the matching
time
>> > > obs_window
>> > > > > > > > >>>>>> from 0 to:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> obs_window = {
>> > > > > > > > >>>>>> beg = 3600;
>> > > > > > > > >>>>>> end = 3600;
>> > > > > > > > >>>>>> }
>> > > > > > > > >>>>>> This tells Point-Stat to match observations
which are
>> > > > exactly
>> > > > > > > > >>>>>> 1-
>> > > > > > > > >>>>>> hour
>> > > > > > > > >>>>>> prior to the valid time of the forecast.
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary
>> from
>> > > 05Z
>> > > > > > > > >>>>>> to
>> > > > > > > > >>>>>> 05Z
>> > > > > > > > >>>>>> instead and then leave the Point-Stat
obs_window
>> setting
>> > > at
>> > > > 0:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> time_summary = {
>> > > > > > > > >>>>>> flag = TRUE;
>> > > > > > > > >>>>>> beg = "050000";
>> > > > > > > > >>>>>> end = "050000";
>> > > > > > > > >>>>>> ...
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> Thanks,
>> > > > > > > > >>>>>> John
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran
- NOAA
>> > > > > > > > >>>>>> Affiliate
>> > > > > > > > >>>>>> <
>> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
output
>> file,
>> > > but
>> > > > > > > > >>>>>>> it's
>> > > > > > > > >>>>>>> empty. Here's the output from point_stat:
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> DEBUG 4:
>> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
>> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
>> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
>> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
>> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
>> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
>> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
>> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
>> > > > observation
>> > > > > > > > >>>>>>> type
>> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
>> interpolation
>> > > > method
>> > > > > > > > >>>>>>> NEAREST(1),
>> > > > > > > > >>>>>>> using 0 pairs.
>> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
>> > > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Have any thoughts about this one?
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Thanks!
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Perry
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry
Shafran -
>> NOAA
>> > > > > > > > >>>>>>> Affiliate
>> > > > > > > > >>>>>>> <
>> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>>> Hi, John,
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> Looks like the changing of the time stamp
from _04
>> to
>> > > _05
>> > > > > > > > >>>>>>>> did
>> > > > > > > > >>>>>>>> the
>> > > > > > > > >>>>>>>> trick as point_stat ran without an issue
with that
>> > > change.
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> Perry
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John
Halley
>> Gotway <
>> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>>> Perry,
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
>> > the
>> > > > > > > > >>>>>>>>> timestamp
>> > > > > > > > >>>>>>>>> of the data from that file. I think
switching the
>> > > > > > > > >>>>>>>>> verification time to
>> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix
it.
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
>> > > > initialization
>> > > > > > > > >>>>>>>>> times
>> > > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
>> > > wgrib2
>> > > > > > > > >>>>>>>>> and
>> > > > > > > > >>>>>>>>> MET commands
>> > > > > > > > >>>>>>>>> to illustrate.
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-
t),
>> > forecast
>> > > > > > > > >>>>>>>>> time
>> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using
>> a
>> > > > sample
>> > > > > > > > >>>>>>>>> data file you
>> > > > > > > > >>>>>>>>> sent to me. But remember I'm still using
the
>> "old"
>> > > > wgrib2
>> > > > > > > > >>>>>>>>> so
>> > > > > > > > >>>>>>>>> I have that
>> > > > > > > > >>>>>>>>> funky forecast hour:
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
>> > > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624
hour ave
>> > > > > > > > >>>>>>>>> fcst:vt=2018070505
>> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
>> > fcst:vt=2018070605
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a
higher
>> > verbosity
>> > > > > > > > >>>>>>>>> level
>> > > > > > > > >>>>>>>>> (-v
>> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of
the
>> data
>> > it
>> > > is
>> > > > > > > > >>>>>>>>> reading:
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> # Read the first record
>> > > > > > > > >>>>>>>>> plot_data_plane
>> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
>> > > > > > > > >>>>>>>>> ...
>> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
>> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
>> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
>> > > > > > > > >>>>>>>>> ...
>> > > > > > > > >>>>>>>>> # Read the second record
>> > > > > > > > >>>>>>>>> plot_data_plane
>> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
>> > > > > > > > >>>>>>>>> ...
>> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
>> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
>> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> Hope that helps.
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>> John
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
Shafran -
>> NOAA
>> > > > > > > > >>>>>>>>> Affiliate <
>> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>>> Hi, John,
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> When I implemented all the changes (and
also set
>> it
>> > up
>> > > > so
>> > > > > > > > >>>>>>>>>> it's
>> > > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
>> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
>> > > > > > > > >>>>>>>>>>
>> > > > > >
>> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
>> > > > > > > > >>>>>>>>>> -v 4
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> But then I got this error:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> DEBUG 2:
>> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> WARNING:
MetGrib2DataFile::data_plane_array() -
>> No
>> > > > > > > > >>>>>>>>>> matching
>> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
>> > > > matching
>> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> ERROR :
>> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
>> requested
>> > > > > > > > >>>>>>>>>> forecast
>> > > > > > > > >>>>>>>>>> data
>> > > > > > > > >>>>>>>>>> found! Exiting...
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the
wgrib2
>> output
>> > for
>> > > > the
>> > > > > > > > >>>>>>>>>> file
>> > > > > > > > >>>>>>>>>> in this case reads:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> 14 (t14a2)
>> > > > > > > > >>>>>>>>>>
>> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > >
>> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
>> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-
16 hour
>> ave
>> > > > fcst:
>> > > > > > > > >>>>>>>>>> 15 (t14a2)
>> > > > > > > > >>>>>>>>>>
>> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You
>> had
>> > > > > > > > >>>>>>>>>> suggested
>> > > > > > > > >>>>>>>>>> that it should be the time stamp of the
2nd day,
>> but
>> > > > maybe
>> > > > > > > > >>>>>>>>>> it needs to be
>> > > > > > > > >>>>>>>>>> the time stamp of the first day?
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
>> elsewhere.
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John
Halley
>> Gotway <
>> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> Perry,
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air
quality vx we
>> > did,
>> > > > we
>> > > > > > > > >>>>>>>>>>> used
>> > > > > > > > >>>>>>>>>>> observation files named:
>> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs
and
>> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> And those files don't exist in the July
data you
>> > just
>> > > > > > > > >>>>>>>>>>> sent.
>> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
>> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
>> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
>> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
>> > aqm.t12z.anowpm.pb.tm00
>> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
>> aqm.t12z.anowpm.pb.tm120
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
>> against
>> > the
>> > > > > > > > >>>>>>>>>>> BUFR
>> > > > > > > > >>>>>>>>>>> or
>> > > > > > > > >>>>>>>>>>> PREPBUFR.
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> Let me just tell you the key points for
the
>> config
>> > > file
>> > > > > > > > >>>>>>>>>>> settings...
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to
pass 2
>> > input
>> > > > > > > > >>>>>>>>>>> files
>> > > > > > > > >>>>>>>>>>> -
>> > > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
>> > > > > > > > >>>>>>>>>>> pb2nc \
>> > > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00
\
>> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
>> > > > > > > > >>>>>>>>>>> PB2NCConfig \
>> > > > > > > > >>>>>>>>>>> -pbfile
>> hourly.20180704/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
>> > time_summary
>> > > > > > > > >>>>>>>>>>> like
>> > > > > > > > >>>>>>>>>>> this:
>> > > > > > > > >>>>>>>>>>> time_summary = {
>> > > > > > > > >>>>>>>>>>> flag = TRUE;
>> > > > > > > > >>>>>>>>>>> beg = "040000";
>> > > > > > > > >>>>>>>>>>> end = "040000";
>> > > > > > > > >>>>>>>>>>> step = 24*3600;
>> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end =
0; }
>> //
>> > > > > > > > >>>>>>>>>>> previous
>> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
>> > > > > > > > >>>>>>>>>>> grib_code = [];
>> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
>> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
>> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every
>> hour
>> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
>> summaries
>> > > when
>> > > > at
>> > > > > > > > >>>>>>>>>>> least
>> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
>> > > > > > > > >>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values
at each
>> > > station
>> > > > > > > > >>>>>>>>>>> over the
>> > > > > > > > >>>>>>>>>>> defined time period. They're identified
in the
>> > > output
>> > > > by
>> > > > > > > > >>>>>>>>>>> the message type:
>> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst"
>> and
>> > > > "obs"
>> > > > > > > > >>>>>>>>>>> sections like this for the max of the 1-
hour
>> data.
>> > > > > > > > >>>>>>>>>>> Instead
>> > > > > > > > >>>>>>>>>>> of specifying
>> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
>> > > > convenient
>> > > > > > > > >>>>>>>>>>> to specify the
>> > > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify
>> the
>> > > > config
>> > > > > > > > >>>>>>>>>>> file, I'd suggest
>> > > > > > > > >>>>>>>>>>> using environment variables:
>> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> fcst = {
>> > > > > > > > >>>>>>>>>>> field = [
>> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
>> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
>> > > > > > > > >>>>>>>>>>> ];
>> > > > > > > > >>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>> obs = {
>> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
>> > > > > > > > >>>>>>>>>>> field = [
>> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
>> message_type =
>> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
>> > > > > > > > >>>>>>>>>>> ];
>> > > > > > > > >>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
Halley
>> Gotway
>> > <
>> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
>> > NOAA
>> > > > > > > > >>>>>>>>>>>> Affiliate <
>> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all
the bc
>> grids
>> > > and
>> > > > > > > > >>>>>>>>>>>>> focus on
>> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time
to copy
>> > files
>> > > > > > > > >>>>>>>>>>>>> over
>> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
>> > > > > > > > >>>>>>>>>>>>> let him work on the files he has for
now and
>> if
>> > he
>> > > > runs
>> > > > > > > > >>>>>>>>>>>>> into problems, I'll
>> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
least the
>> > 24-hr
>> > > > > > > > >>>>>>>>>>>>> max/avg files).
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
>> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
>> grids...not
>> > > for
>> > > > > > > > >>>>>>>>>>>>>> bias
>> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
>> > > safe,
>> > > > > > > > >>>>>>>>>>>>>> you
>> > > > > > > > >>>>>>>>>>>>>> may want to provide
>> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran -
NOAA
>> > > Affiliate
>> > > > > > > > >>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over
to Theia
>> > for
>> > > > John
>> > > > > > > > >>>>>>>>>>>>>> to do
>> > > > > > > > >>>>>>>>>>>>>> some testing some production files.
Do you
>> > think
>> > > I
>> > > > > > > > >>>>>>>>>>>>>> should use parallels
>> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files
are OK
>> for
>> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
>> > > > > > > > >>>>>>>>>>>>>> for AQ?
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think
that the
>> ops
>> > > > files
>> > > > > > > > >>>>>>>>>>>>>> should
>> > > > > > > > >>>>>>>>>>>>>> be OK for now.
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen
>> <
>> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about
the
>> > > parallels.
>> > > > > > > > >>>>>>>>>>>>>>> We
>> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily
max hour
>> > > labels
>> > > > > > > > >>>>>>>>>>>>>>> but
>> > > > > > > > >>>>>>>>>>>>>>> management did not
>> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19
CMAQ
>> > upgrade.
>> > > > > > > > >>>>>>>>>>>>>>> Jeff
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
>> > > > Affiliate
>> > > > > > > > >>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have
the
>> > correct
>> > > > time
>> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send
>> John
>> > > > over
>> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
>> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has
any
>> other
>> > > > > > > > >>>>>>>>>>>>>>> versions
>> > > > > > > > >>>>>>>>>>>>>>> of
>> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have
- but
>> > they
>> > > > can
>> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
>> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new
>> does
>> > > over
>> > > > > > > > >>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
>> jeff.mcqueen <
>> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you
may
>> want
>> > to
>> > > > use
>> > > > > > > > >>>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
>> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
>> > > Jianping
>> > > > > > > > >>>>>>>>>>>>>>>> also
>> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it
but
>> > > management
>> > > > > > > > >>>>>>>>>>>>>>>> did
>> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
>> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do
so
>> with
>> > the
>> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
>> > > > > > > > >>>>>>>>>>>>>>>> implementation.
>> > > > > > > > >>>>>>>>>>>>>>>> Jeff
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
>> > > > Affiliate
>> > > > > > > > >>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
>> conversation.
>> > I
>> > > > > > > > >>>>>>>>>>>>>>>> think
>> > > > > > > > >>>>>>>>>>>>>>>> I
>> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the
timestamps some
>> > time
>> > > > ago,
>> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
>> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
>> Jeff. I
>> > > > think
>> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
>> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not
too
>> > entirely
>> > > > > > > > >>>>>>>>>>>>>>>> sure.
>> > > > > > > > >>>>>>>>>>>>>>>> But he would have
>> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct
and how
>> to
>> > > > > > > > >>>>>>>>>>>>>>>> approach
>> > > > > > > > >>>>>>>>>>>>>>>> this.
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time
stamp
>> issues
>> > > John
>> > > > > > > > >>>>>>>>>>>>>>>> talks
>> > > > > > > > >>>>>>>>>>>>>>>> about?
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
>> > > > Gotway <
>> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all
squared away
>> to
>> > > > > > > > >>>>>>>>>>>>>>>>> compute
>> > > > > > > > >>>>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values
for 1-
>> and
>> > > 8-
>> > > > > > > > >>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
>> > \
>> > > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
>> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
>> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
>> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like
this:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
>> > > > > \
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
>> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
PointStatConfig_AIRNOW \
>> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
>> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1
versus
>> COPO/A1,
>> > > for
>> > > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
>> > region
>> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
>> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
>> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147
pairs.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
>> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
>> contents
>> > of
>> > > > > > > > >>>>>>>>>>>>>>>>> this
>> > > > > > > > >>>>>>>>>>>>>>>>> "max"
>> > > > > > > > >>>>>>>>>>>>>>>>> file:
>> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
>> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour
ave
>> fcst:
>> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
sigma
>> > > > level:17-40
>> > > > > > > > >>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
>> > > testing
>> > > > > > > > >>>>>>>>>>>>>>>>> revealed
>> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
>> > > hours
>> > > > 17
>> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
>> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
>> > > better.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours
are in
>> the
>> > > same
>> > > > > > > > >>>>>>>>>>>>>>>>> file, we
>> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to
tell it
>> which
>> > > one
>> > > > > > > > >>>>>>>>>>>>>>>>> to
>> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
>> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
>> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
>> > > > > > > > >>>>>>>>>>>>>>>>> field = [
>> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
>> > > > lead_time
>> > > > > > > > >>>>>>>>>>>>>>>>> =
>> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
>> > > > > > > > >>>>>>>>>>>>>>>>> ];
>> > > > > > > > >>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
>> timestamps
>> > are
>> > > > off
>> > > > > > > > >>>>>>>>>>>>>>>>> by 1
>> > > > > > > > >>>>>>>>>>>>>>>>> hour.
>> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a
41 hour
>> > lead
>> > > > > > > > >>>>>>>>>>>>>>>>> time
>> > > > > > > > >>>>>>>>>>>>>>>>> =
>> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
>> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a
24-hour
>> max
>> > > > value
>> > > > > > > > >>>>>>>>>>>>>>>>> from
>> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
valid at
>> 05Z
>> > > > while
>> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
>> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
>> Point-Stat
>> > > > config
>> > > > > > > > >>>>>>>>>>>>>>>>> file by
>> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look
back 1
>> hour:
>> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
>> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
>> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
>> > > > > > > > >>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM
hsoh
>> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
>> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files
and was
>> > > fixed.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
>> > > wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
>> summaries,
>> > > > but I
>> > > > > > > > >>>>>>>>>>>>>>>>>> found
>> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a
look on
>> > dakota
>> > > > > > > > >>>>>>>>>>>>>>>>>> here:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > /d3/projects/MET/MET_Help/shafran_data_20180626
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc
to
>> compute
>> > > 24-
>> > > > > > > > >>>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped
>> the
>> > > > output
>> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
>> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands
>> to
>> > > count
>> > > > > > > > >>>>>>>>>>>>>>>>>> up
>> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
>> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
>> > > pb2nc_output.txt
>> > > > |
>> > > > > > > > >>>>>>>>>>>>>>>>>> wc
>> > > > > > > > >>>>>>>>>>>>>>>>>> -l
>> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
>> > > pb2nc_output.txt
>> > > > |
>> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
>> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
>> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
>> > pb2nc_output.txt |
>> > > > wc
>> > > > > > > > >>>>>>>>>>>>>>>>>> -l
>> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
>> > pb2nc_output.txt |
>> > > > > > > > >>>>>>>>>>>>>>>>>> sort
>> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
>> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
>> meaning
>> > > that
>> > > > > > > > >>>>>>>>>>>>>>>>>> all
>> > > > > > > > >>>>>>>>>>>>>>>>>> of
>> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
unique.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
>> > > 6,555
>> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
>> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations
>> are
>> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID
in that
>> > file
>> > > is
>> > > > > > > > >>>>>>>>>>>>>>>>>> named
>> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
>> > grep
>> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180527_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180527_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
>> repeated in
>> > > the
>> > > > > > > > >>>>>>>>>>>>>>>>>> last two
>> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
>> > > > duplicates
>> > > > > > > > >>>>>>>>>>>>>>>>>> are
>> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM
hsoh
>> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
>> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
>> > > wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's
no
>> reason
>> > why
>> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
>> > > > > > > > >>>>>>>>>>>>>>>>>>> have
>> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
>> > > across 2
>> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
>> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from
which we
>> > could
>> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
>> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard
>> Soh
>> > <
>> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
>> > > > (550.319
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
minutes.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will
not
>> check
>> > in
>> > > > the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
files.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
>> > content
>> > > > of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
>> > > summary
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple
BURF
>> > inputs.
>> > > I
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files
>> and
>> > > > will
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
Halley
>> Gotway
>> > > > wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this
will
>> change
>> > > the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case
not
>> > > currently
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
>> > Soh
>> > > <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
>> > > summary
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to
SVN.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
>> /d3/personal/hsoh/data/pb2nc_summary/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52
PM John
>> > > Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates
to the
>> > logic
>> > > > for
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
>> compute
>> > > > 24-hour
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
>> > > > counts:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
>> > currently)
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript
>> to
>> > > dump
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
>> > > string
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem
is in
>> PB2NC
>> > > not
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
summaries... or
>> in
>> > the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54
PM John
>> > > Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41
PM
>> Howard
>> > > > Soh <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291
observations from
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> aqm.20180527/aqm.t12z.prepbufr.tm00
>> > > and
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
>> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for
the obs
>> > > data.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at
2:28 PM
>> John
>> > > > Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
Could you
>> > > please
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
>> > PB2NC.
>> > > > I
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc
PB2NCConfig \
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
request
>> > > 24-hourly
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z,
like
>> this:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
>> > end =
>> > > > 0;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
>> > but
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is
not
>> what
>> > > you'd
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
>> > > > "AIRNOW_MAX_240000"
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
referenced.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
>> > caused
>> > > by
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I
need 2
>> of
>> > > them
>> > > > to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to
04Z
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a
look?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:44 PM
>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
>> > > PrepBUFR
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
>> compute a
>> > 24
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
>> > week.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:40 PM
>> > hsoh
>> > > <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script
is
>> > > reasonable
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
>> > data
>> > > in
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
index of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file
>> was
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from
pb2nc
>> > without
>> > > > any
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
>> > > headers
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
>> > NetCDF
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(7295
>> > > currently)
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
value
>> from 0
>> > > to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message
type for
>> > > > summary)
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
>> variable.
>> > > The
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger
>> than
>> > > > 3655.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to
be OK.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > >
>> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEDIAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEDIAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM,
John
>> Halley
>> > > > Gotway
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a
look on
>> > dakota
>> > > > in
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to
dump it
>> to
>> > > > ascii:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
>> > > > /d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
>> > pb2nc_output.txt
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error
but lists
>> > all
>> > > of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But
>> some
>> > > of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
>> evidenced
>> > by
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
>> > > > pb2nc_output.nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
indexing or
>> > table
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Jun 07 05:45:06 2019
Hi, John,
Thanks!
Perry
On Thu, Jun 6, 2019 at 6:21 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry (and Howard),
>
> I isolated this issue down and wrote up a GitHub issue describing
the
> problem:
> https://github.com/NCAR/MET/issues/1137
>
> Howard, please take a look and let me know if you need any help
further
> debugging this.
>
> Thanks,
> John
>
> On Thu, Jun 6, 2019 at 3:21 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Perry,
> >
> > I was able to replicate this behavior both on WCOSS and locally.
I
> > believe it's a bug in PB2NC. I'm trying to narrow it down and
will then
> > hand it off to Howard to debug. Thanks for identifying this
issue.
> >
> > John
> >
> > On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >>
> >> Hi, John,
> >>
> >> Well, it does appear that I got the same error with MET 8.1 as I
did
> with
> >> the previous version:
> >>
> >> ERROR :
> >> ERROR : StringArray::operator[](int) const -> range check error!
> >> ERROR :
> >>
> >> Perry
> >>
> >> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Perry,
> >> >
> >> > Did you ever get this issue resolved from 1 month ago? Or
should I
> log
> >> on
> >> > to WCOSS and take a look?
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Hi, John,
> >> > >
> >> > > Well, I am running the script in batch so I don't control
what node
> it
> >> > > gets. I can tell you that the working directory is here:
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > >
> >> > > And the output log file is here:
> >> > >
/ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688, which
> >> > should
> >> > > help you figure out how the thing is run.
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Perry
> >> > >
> >> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> >> > > met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Perry,
> >> > > >
> >> > > > We can log on to wcoss and test there instead. Yes, pb2nc
does
> >> seem to
> >> > > be
> >> > > > the culprit. Can you point us the wcoss node you're using
and the
> >> > > working
> >> > > > directory there? We basically need to recreate your call
to pb2nc
> >> to
> >> > try
> >> > > > to replicate the behavior.
> >> > > >
> >> > > > Thanks,
> >> > > > John
> >> > > >
> >> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via
RT <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >> > > > >
> >> > > > > Hi, David, John,
> >> > > > >
> >> > > > > I am not running on Theia. In fact I think it would take
quite
> a
> >> bit
> >> > > of
> >> > > > > effort to transition this thing to Theia, and I'll need
to be
> >> running
> >> > > > this
> >> > > > > verification on prod data anyway, which is only on WCOSS.
> >> > > > >
> >> > > > > However, if you wish to troubleshoot on Theia, you should
do so,
> >> and
> >> > I
> >> > > > > think you have all you need to be able to do so, right?
> >> > > > >
> >> > > > > It would seem that the error on my side is in PB2NC, so
I'll
> >> probably
> >> > > > need
> >> > > > > to make sure I'm running what you're running.
> >> > > > >
> >> > > > > Thanks!
> >> > > > >
> >> > > > > Perry
> >> > > > >
> >> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> >> > > met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Perry -
> >> > > > > > Are you running on theia?
> >> > > > > > John suggests that we try to troubleshoot there ...
> >> > > > > > thanks,
> >> > > > > > David
> >> > > > > >
> >> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> >> > > > > > > Hi Perry -
> >> > > > > > >
> >> > > > > > > An update on where we are on this issue:
> >> > > > > > > Using the prepda.2019042312.nc file you provided we
indeed
> >> > > reproduce
> >> > > > > > > the point_stat error.
> >> > > > > > > The point_stat error is due to only a single entry
the valid
> >> time
> >> > > > > > > table,
> >> > > > > > > e.g. from ncdump -v hdr_vld_table
> >> > > > > > > hdr_vld_table =
> >> > > > > > > "20190423_050000",
> >> > > > > > > while the hdr_vld indices in that file are hdr_vld =
0, 33,
> >> 33,
> >> > ...
> >> > > > > > > which are all out of range except the first.
> >> > > > > > >
> >> > > > > > > If I run pb2nc here with your PREPBUFR data and your
> >> > > PB2NCConfig_aqm
> >> > > > > > > config file, however,
> >> > > > > > > I generate a different prepda.2019042312.nc file that
runs
> >> > through
> >> > > > to
> >> > > > > > > completion with point_stat.
> >> > > > > > > Its valid time table:
> >> > > > > > >
> >> > > > > > > hdr_vld_table =
> >> > > > > > > "20190423_030000",
> >> > > > > > > "20190423_010000",
> >> > > > > > > "20190423_020000",
> >> > > > > > > "20190423_230000",
> >> > > > > > > "20190423_150000",
> >> > > > > > > "20190423_090000",
> >> > > > > > > "20190423_070000",
> >> > > > > > > "20190423_060000",
> >> > > > > > > "20190423_210000",
> >> > > > > > > "20190423_050000",
> >> > > > > > > "20190423_170000",
> >> > > > > > > "20190423_100000",
> >> > > > > > > "20190423_200000",
> >> > > > > > > "20190423_140000",
> >> > > > > > > "20190423_130000",
> >> > > > > > > "20190423_160000",
> >> > > > > > > "20190423_040000",
> >> > > > > > > "20190423_180000",
> >> > > > > > > "20190423_080000",
> >> > > > > > > "20190423_120000",
> >> > > > > > > "20190423_110000",
> >> > > > > > > "20190423_220000",
> >> > > > > > > "20190423_190000",
> >> > > > > > > "20190424_000000" ;
> >> > > > > > > }
> >> > > > > > >
> >> > > > > > > So it seems pb2nc is running correctly here, but not
on your
> >> > side.
> >> > > > > > > Let me discuss with John the best way to proceed ...
> >> > > > > > > thanks,
> >> > > > > > > David
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
wrote:
> >> > > > > > > > Hi, everyone,
> >> > > > > > > >
> >> > > > > > > > I am attaching this to this long thread regarding
the
> daily
> >> max
> >> > > of
> >> > > > > > > > ozone
> >> > > > > > > > verification, so we remember what we were doing
back then.
> >> > > > > > > >
> >> > > > > > > > I am now running this again using MET 8.0, and
something
> >> seems
> >> > to
> >> > > > not
> >> > > > > > > > be
> >> > > > > > > > working when I run point_stat. I get this error
when
> >> running a
> >> > > > daily
> >> > > > > > > > max
> >> > > > > > > > of ozone:
> >> > > > > > > >
> >> > > > > > > > ERROR :
> >> > > > > > > > ERROR : StringArray::operator[](int) const ->
range check
> >> > error!
> >> > > > > > > > ERROR :
> >> > > > > > > >
> >> > > > > > > > I am using my own scripts to run this using MET, so
this
> is
> >> not
> >> > > > using
> >> > > > > > > > METplus.
> >> > > > > > > >
> >> > > > > > > > You can find a log file of this here:
> >> > > > > > > >
> >> > > > >
> >> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> >> > > > > > > > It
> >> > > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc
code is
> >> > > working
> >> > > > > > > > fine, I
> >> > > > > > > > believe.
> >> > > > > > > >
> >> > > > > > > > The script I am using to run this is here:
> >> > > > > > > >
> >> >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> >> > > > > > > >
> >> > > > > > > > The prepbufr files are:
> >> > > > > > > >
> >> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
> and
> >> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> >> > > > > > > >
> >> > > > > > > > The model file is here:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > What could be causing this error?
> >> > > > > > > >
> >> > > > > > > > Thanks!
> >> > > > > > > >
> >> > > > > > > > Perry
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<
> >> > > > johnhg at ucar.edu>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Great, glad that did the trick.
> >> > > > > > > > >
> >> > > > > > > > > John
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran -
NOAA
> >> > Affiliate
> >> > > <
> >> > > > > > > > > perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Well this worked! And Jeff confirmed to me that
the
> 8-hr
> >> > > > average
> >> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z
like in
> >> the
> >> > > 1-hr
> >> > > > > > > > >> avg.
> >> > > > > > > > >>
> >> > > > > > > > >> Thanks!
> >> > > > > > > > >>
> >> > > > > > > > >> Perry
> >> > > > > > > > >>
> >> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley
Gotway
> >> > > > > > > > >> <johnhg at ucar.edu>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >>
> >> > > > > > > > >>> Perry,
> >> > > > > > > > >>>
> >> > > > > > > > >>> I think the problem is in your setting of
> >> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
> >> > > > > > > > >>>
> >> > > > > > > > >>> Here's your wgrib2 output:
> >> > > > > > > > >>> wgrib2 oz_DAY1
> >> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour
> ave
> >> > fcst:
> >> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-
1-22
> hour
> >> > ave
> >> > > > > > > > >>> fcst:
> >> > > > > > > > >>>
> >> > > > > > > > >>> The "d=2018081112" tells you the model
initialization
> >> time.
> >> > > > The
> >> > > > > > > > >>> valid
> >> > > > > > > > >>> time = init time + forecast time. So the
OZMAX1 data
> is
> >> > > valid
> >> > > > 16
> >> > > > > > > > >>> hours
> >> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
later.
> >> Rerun
> >> > > > that
> >> > > > > > > > >>> wgrib2
> >> > > > > > > > >>> command using the "-vt" argument to list the
> >> verification
> >> > > time.
> >> > > > > > > > >>>
> >> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data
because that
> >> data
> >> > > > isn't
> >> > > > > > > > >>> valid
> >> > > > > > > > >>> at the time you're requesting.
> >> > > > > > > > >>>
> >> > > > > > > > >>> John
> >> > > > > > > > >>>
> >> > > > > > > > >>>
> >> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran -
NOAA
> >> > > Affiliate
> >> > > > <
> >> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>
> >> > > > > > > > >>>> Hi, John,
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I haven't been able to work on this for
awhile, but
> now
> >> > I'm
> >> > > > > > > > >>>> working on
> >> > > > > > > > >>>> it again but seem to be running into issues.
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I have a file that has both the daily max for
1-hr
> >> ozone
> >> > and
> >> > > > the
> >> > > > > > > > >>>> daily
> >> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks
like
> this:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> 9 (t10a2)
> >> > > > > > > > >>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>> >
> >> > > > > > > > >>>> wgrib2 oz_DAY1
> >> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour ave
> >> fcst:
> >> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-
22 hour
> >> ave
> >> > > > fcst:
> >> > > > > > > > >>>> 10 (t10a2)
> >> > > > > > > > >>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>> >
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> As you can see they have different forecast
times.
> But
> >> > > when I
> >> > > > > > > > >>>> run
> >> > > > > > > > >>>> point_stat, I get this output:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>>
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array()
- No
> >> > matching
> >> > > > > > > > >>>> records
> >> > > > > > > > >>>> found for 'OZMAX8/L1'
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no
fields
> >> matching
> >> > > > > > > > >>>> OZMAX8/L1
> >> > > > > > > > >>>> found in file: oz_DAY1
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Here is the relevant portion of the point_stat
config
> >> > file:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> fcst = {
> >> > > > > > > > >>>> field = [
> >> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> >> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> >> > > > > > > > >>>> ];
> >> > > > > > > > >>>> }
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> obs = {
> >> > > > > > > > >>>> convert(x) = x * 10^9;
> >> > > > > > > > >>>> field = [
> >> > > > > > > > >>>> { name = "COPO"; level = "A1";
message_type =
> >> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
> >> > > > > > > > >>>> { name = "COPO"; level = "A8";
message_type =
> >> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
> >> > > > > > > > >>>> ];
> >> > > > > > > > >>>> }
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I don't remember if I had this working
previously for
> >> both
> >> > > the
> >> > > > > > > > >>>> 1hr
> >> > > > > > > > >>>> and
> >> > > > > > > > >>>> the 8hr ozone, but apparently there is some
sort of
> >> issue
> >> > > for
> >> > > > > > > > >>>> the
> >> > > > > > > > >>>> 8hr
> >> > > > > > > > >>>> ozone. Have any thoughts?
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Thanks!
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Perry
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>> <
> >> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>>> I chose option 2, and that seems to work - I
have a
> >> .stat
> >> > > > file
> >> > > > > > > > >>>>> with
> >> > > > > > > > >>>>> output. Now I'll need to do some work here
to see
> if
> >> the
> >> > > > > > > > >>>>> output
> >> > > > > > > > >>>>> looks OK,
> >> > > > > > > > >>>>> and to configure the verification so it
verifies
> >> OZMAX1
> >> > and
> >> > > > > > > > >>>>> OZMAX8.
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> I also wonder if this would be easily
modified for
> PM.
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> Perry
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> >> > > > > > > > >>>>> <johnhg at ucar.edu>
> >> > > > > > > > >>>>> wrote:
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>>> Perry,
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute
time
> >> > summaries
> >> > > > > > > > >>>>>> that
> >> > > > > > > > >>>>>> are
> >> > > > > > > > >>>>>> valid at 04Z. And you're comparing them to
> forecasts
> >> > that
> >> > > > are
> >> > > > > > > > >>>>>> purportedly
> >> > > > > > > > >>>>>> valid at 05Z. And that's the reason why you
have
> so
> >> > many
> >> > > > obs
> >> > > > > > > > >>>>>> rejected due
> >> > > > > > > > >>>>>> to the valid time.
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> It's up to you guys to decide what the
"correct"
> way
> >> of
> >> > > > > > > > >>>>>> handling
> >> > > > > > > > >>>>>> this
> >> > > > > > > > >>>>>> is... but here are a couple of options:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> (1) You could use the existing setup and
address
> this
> >> > > logic
> >> > > > in
> >> > > > > > > > >>>>>> Point-Stat config file. Change the matching
time
> >> > > obs_window
> >> > > > > > > > >>>>>> from 0 to:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> obs_window = {
> >> > > > > > > > >>>>>> beg = 3600;
> >> > > > > > > > >>>>>> end = 3600;
> >> > > > > > > > >>>>>> }
> >> > > > > > > > >>>>>> This tells Point-Stat to match observations
which
> are
> >> > > > exactly
> >> > > > > > > > >>>>>> 1-
> >> > > > > > > > >>>>>> hour
> >> > > > > > > > >>>>>> prior to the valid time of the forecast.
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary
> >> from
> >> > > 05Z
> >> > > > > > > > >>>>>> to
> >> > > > > > > > >>>>>> 05Z
> >> > > > > > > > >>>>>> instead and then leave the Point-Stat
obs_window
> >> setting
> >> > > at
> >> > > > 0:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> time_summary = {
> >> > > > > > > > >>>>>> flag = TRUE;
> >> > > > > > > > >>>>>> beg = "050000";
> >> > > > > > > > >>>>>> end = "050000";
> >> > > > > > > > >>>>>> ...
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> Thanks,
> >> > > > > > > > >>>>>> John
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry
Shafran -
> NOAA
> >> > > > > > > > >>>>>> Affiliate
> >> > > > > > > > >>>>>> <
> >> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
output
> >> file,
> >> > > but
> >> > > > > > > > >>>>>>> it's
> >> > > > > > > > >>>>>>> empty. Here's the output from point_stat:
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> DEBUG 4:
> >> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
> >> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> >> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> >> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> >> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> >> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> >> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> >> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1, for
> >> > > > observation
> >> > > > > > > > >>>>>>> type
> >> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
> >> interpolation
> >> > > > method
> >> > > > > > > > >>>>>>> NEAREST(1),
> >> > > > > > > > >>>>>>> using 0 pairs.
> >> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Looks like most rejections were due to
valid time.
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Have any thoughts about this one?
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Thanks!
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Perry
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry
Shafran -
> >> NOAA
> >> > > > > > > > >>>>>>> Affiliate
> >> > > > > > > > >>>>>>> <
> >> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>> Hi, John,
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Looks like the changing of the time stamp
from
> _04
> >> to
> >> > > _05
> >> > > > > > > > >>>>>>>> did
> >> > > > > > > > >>>>>>>> the
> >> > > > > > > > >>>>>>>> trick as point_stat ran without an issue
with
> that
> >> > > change.
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Perry
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John
Halley
> >> Gotway <
> >> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>>> Perry,
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
> requesting
> >> > the
> >> > > > > > > > >>>>>>>>> timestamp
> >> > > > > > > > >>>>>>>>> of the data from that file. I think
switching
> the
> >> > > > > > > > >>>>>>>>> verification time to
> >> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix
it.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on
the
> >> > > > initialization
> >> > > > > > > > >>>>>>>>> times
> >> > > > > > > > >>>>>>>>> and valid times for this data. Here's
some
> useful
> >> > > wgrib2
> >> > > > > > > > >>>>>>>>> and
> >> > > > > > > > >>>>>>>>> MET commands
> >> > > > > > > > >>>>>>>>> to illustrate.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time
(-t),
> >> > forecast
> >> > > > > > > > >>>>>>>>> time
> >> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it
is
> using
> >> a
> >> > > > sample
> >> > > > > > > > >>>>>>>>> data file you
> >> > > > > > > > >>>>>>>>> sent to me. But remember I'm still using
the
> >> "old"
> >> > > > wgrib2
> >> > > > > > > > >>>>>>>>> so
> >> > > > > > > > >>>>>>>>> I have that
> >> > > > > > > > >>>>>>>>> funky forecast hour:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> >> > > > > > > > >>>>>>>>>
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624
hour
> ave
> >> > > > > > > > >>>>>>>>> fcst:vt=2018070505
> >> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> >> > fcst:vt=2018070605
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a
higher
> >> > verbosity
> >> > > > > > > > >>>>>>>>> level
> >> > > > > > > > >>>>>>>>> (-v
> >> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info
of the
> >> data
> >> > it
> >> > > is
> >> > > > > > > > >>>>>>>>> reading:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> # Read the first record
> >> > > > > > > > >>>>>>>>> plot_data_plane
> >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> >> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> # Read the second record
> >> > > > > > > > >>>>>>>>> plot_data_plane
> >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> >> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Hope that helps.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>> John
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
Shafran -
> >> NOAA
> >> > > > > > > > >>>>>>>>> Affiliate <
> >> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Hi, John,
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> When I implemented all the changes (and
also
> set
> >> it
> >> > up
> >> > > > so
> >> > > > > > > > >>>>>>>>>> it's
> >> > > > > > > > >>>>>>>>>> reading the beta6 version), I used this
> command:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> >> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1
prepda.2018070912.nc
> >> > > > > > > > >>>>>>>>>>
> >> > > > > >
> >> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> >> > > > > > > > >>>>>>>>>> -v 4
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> But then I got this error:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> DEBUG 2:
> >> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING:
MetGrib2DataFile::data_plane_array() -
> >> No
> >> > > > > > > > >>>>>>>>>> matching
> >> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() ->
no
> fields
> >> > > > matching
> >> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> ERROR :
> >> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() ->
no
> >> requested
> >> > > > > > > > >>>>>>>>>> forecast
> >> > > > > > > > >>>>>>>>>> data
> >> > > > > > > > >>>>>>>>>> found! Exiting...
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the
wgrib2
> >> output
> >> > for
> >> > > > the
> >> > > > > > > > >>>>>>>>>> file
> >> > > > > > > > >>>>>>>>>> in this case reads:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> 14 (t14a2)
> >> > > > > > > > >>>>>>>>>>
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > >
> >> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> >> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-
7-16
> hour
> >> ave
> >> > > > fcst:
> >> > > > > > > > >>>>>>>>>> 15 (t14a2)
> >> > > > > > > > >>>>>>>>>>
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be
20180708_04? You
> >> had
> >> > > > > > > > >>>>>>>>>> suggested
> >> > > > > > > > >>>>>>>>>> that it should be the time stamp of the
2nd
> day,
> >> but
> >> > > > maybe
> >> > > > > > > > >>>>>>>>>> it needs to be
> >> > > > > > > > >>>>>>>>>> the time stamp of the first day?
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
> >> elsewhere.
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John
Halley
> >> Gotway <
> >> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> Perry,
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air
quality vx
> we
> >> > did,
> >> > > > we
> >> > > > > > > > >>>>>>>>>>> used
> >> > > > > > > > >>>>>>>>>>> observation files named:
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs
and
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> And those files don't exist in the July
data
> you
> >> > just
> >> > > > > > > > >>>>>>>>>>> sent.
> >> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names
like:
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> >> > aqm.t12z.anowpm.pb.tm00
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
> >> aqm.t12z.anowpm.pb.tm120
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually
verifying
> >> against
> >> > the
> >> > > > > > > > >>>>>>>>>>> BUFR
> >> > > > > > > > >>>>>>>>>>> or
> >> > > > > > > > >>>>>>>>>>> PREPBUFR.
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> Let me just tell you the key points for
the
> >> config
> >> > > file
> >> > > > > > > > >>>>>>>>>>> settings...
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to
pass 2
> >> > input
> >> > > > > > > > >>>>>>>>>>> files
> >> > > > > > > > >>>>>>>>>>> -
> >> > > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> >> > > > > > > > >>>>>>>>>>> pb2nc \
> >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00 \
> >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc
> \
> >> > > > > > > > >>>>>>>>>>> PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>> -pbfile
> >> hourly.20180704/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up
the
> >> > time_summary
> >> > > > > > > > >>>>>>>>>>> like
> >> > > > > > > > >>>>>>>>>>> this:
> >> > > > > > > > >>>>>>>>>>> time_summary = {
> >> > > > > > > > >>>>>>>>>>> flag = TRUE;
> >> > > > > > > > >>>>>>>>>>> beg = "040000";
> >> > > > > > > > >>>>>>>>>>> end = "040000";
> >> > > > > > > > >>>>>>>>>>> step = 24*3600;
> >> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end
= 0; }
> >> //
> >> > > > > > > > >>>>>>>>>>> previous
> >> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> >> > > > > > > > >>>>>>>>>>> grib_code = [];
> >> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> >> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> >> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect
obs every
> >> hour
> >> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
> >> summaries
> >> > > when
> >> > > > at
> >> > > > > > > > >>>>>>>>>>> least
> >> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values
at
> each
> >> > > station
> >> > > > > > > > >>>>>>>>>>> over the
> >> > > > > > > > >>>>>>>>>>> defined time period. They're
identified in
> the
> >> > > output
> >> > > > by
> >> > > > > > > > >>>>>>>>>>> the message type:
> >> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup
the
> "fcst"
> >> and
> >> > > > "obs"
> >> > > > > > > > >>>>>>>>>>> sections like this for the max of the
1-hour
> >> data.
> >> > > > > > > > >>>>>>>>>>> Instead
> >> > > > > > > > >>>>>>>>>>> of specifying
> >> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may
be more
> >> > > > convenient
> >> > > > > > > > >>>>>>>>>>> to specify the
> >> > > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify
> >> the
> >> > > > config
> >> > > > > > > > >>>>>>>>>>> file, I'd suggest
> >> > > > > > > > >>>>>>>>>>> using environment variables:
> >> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> fcst = {
> >> > > > > > > > >>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> >> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> >> > > > > > > > >>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>> obs = {
> >> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> >> > > > > > > > >>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
> >> message_type =
> >> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> >> > > > > > > > >>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
Halley
> >> Gotway
> >> > <
> >> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
> Shafran -
> >> > NOAA
> >> > > > > > > > >>>>>>>>>>>> Affiliate <
> >> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all
the bc
> >> grids
> >> > > and
> >> > > > > > > > >>>>>>>>>>>>> focus on
> >> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time
to
> copy
> >> > files
> >> > > > > > > > >>>>>>>>>>>>> over
> >> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
> >> > > > > > > > >>>>>>>>>>>>> let him work on the files he has for
now and
> >> if
> >> > he
> >> > > > runs
> >> > > > > > > > >>>>>>>>>>>>> into problems, I'll
> >> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
least
> the
> >> > 24-hr
> >> > > > > > > > >>>>>>>>>>>>> max/avg files).
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
> jeff.mcqueen <
> >> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
> >> grids...not
> >> > > for
> >> > > > > > > > >>>>>>>>>>>>>> bias
> >> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus
grids, To
> be
> >> > > safe,
> >> > > > > > > > >>>>>>>>>>>>>> you
> >> > > > > > > > >>>>>>>>>>>>>> may want to provide
> >> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran -
NOAA
> >> > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over
to
> Theia
> >> > for
> >> > > > John
> >> > > > > > > > >>>>>>>>>>>>>> to do
> >> > > > > > > > >>>>>>>>>>>>>> some testing some production files.
Do you
> >> > think
> >> > > I
> >> > > > > > > > >>>>>>>>>>>>>> should use parallels
> >> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops
files are
> OK
> >> for
> >> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> >> > > > > > > > >>>>>>>>>>>>>> for AQ?
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think
that
> the
> >> ops
> >> > > > files
> >> > > > > > > > >>>>>>>>>>>>>> should
> >> > > > > > > > >>>>>>>>>>>>>> be OK for now.
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
> jeff.mcqueen
> >> <
> >> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about
the
> >> > > parallels.
> >> > > > > > > > >>>>>>>>>>>>>>> We
> >> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily
max
> hour
> >> > > labels
> >> > > > > > > > >>>>>>>>>>>>>>> but
> >> > > > > > > > >>>>>>>>>>>>>>> management did not
> >> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19
CMAQ
> >> > upgrade.
> >> > > > > > > > >>>>>>>>>>>>>>> Jeff
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels
have the
> >> > correct
> >> > > > time
> >> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if
I send
> >> John
> >> > > > over
> >> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> >> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has
any
> >> other
> >> > > > > > > > >>>>>>>>>>>>>>> versions
> >> > > > > > > > >>>>>>>>>>>>>>> of
> >> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already
have -
> but
> >> > they
> >> > > > can
> >> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
> >> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new
> >> does
> >> > > over
> >> > > > > > > > >>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
> >> jeff.mcqueen <
> >> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new,
you may
> >> want
> >> > to
> >> > > > use
> >> > > > > > > > >>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output
(para5).
> >> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> >> > > Jianping
> >> > > > > > > > >>>>>>>>>>>>>>>> also
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it
but
> >> > > management
> >> > > > > > > > >>>>>>>>>>>>>>>> did
> >> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> >> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to
do so
> >> with
> >> > the
> >> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
> >> > > > > > > > >>>>>>>>>>>>>>>> implementation.
> >> > > > > > > > >>>>>>>>>>>>>>>> Jeff
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
> >> conversation.
> >> > I
> >> > > > > > > > >>>>>>>>>>>>>>>> think
> >> > > > > > > > >>>>>>>>>>>>>>>> I
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the
timestamps
> some
> >> > time
> >> > > > ago,
> >> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or
with
> >> Jeff. I
> >> > > > think
> >> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> >> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not
too
> >> > entirely
> >> > > > > > > > >>>>>>>>>>>>>>>> sure.
> >> > > > > > > > >>>>>>>>>>>>>>>> But he would have
> >> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct
and
> how
> >> to
> >> > > > > > > > >>>>>>>>>>>>>>>> approach
> >> > > > > > > > >>>>>>>>>>>>>>>> this.
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time
stamp
> >> issues
> >> > > John
> >> > > > > > > > >>>>>>>>>>>>>>>> talks
> >> > > > > > > > >>>>>>>>>>>>>>>> about?
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM,
John
> Halley
> >> > > > Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all
squared
> away
> >> to
> >> > > > > > > > >>>>>>>>>>>>>>>>> compute
> >> > > > > > > > >>>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average
values for
> 1-
> >> and
> >> > > 8-
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >> > \
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like
this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> >> > > > > \
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
PointStatConfig_AIRNOW
> \
> >> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> >> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1
versus
> >> COPO/A1,
> >> > > for
> >> > > > > > > > >>>>>>>>>>>>>>>>> observation type
AIRNOW_MAX_240000, over
> >> > region
> >> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
> >> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> >> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147
pairs.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> >> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes
the
> >> contents
> >> > of
> >> > > > > > > > >>>>>>>>>>>>>>>>> this
> >> > > > > > > > >>>>>>>>>>>>>>>>> "max"
> >> > > > > > > > >>>>>>>>>>>>>>>>> file:
> >> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> >> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618
hour ave
> >> fcst:
> >> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
sigma
> >> > > > level:17-40
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty
funny. But
> >> > > testing
> >> > > > > > > > >>>>>>>>>>>>>>>>> revealed
> >> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
> forecast
> >> > > hours
> >> > > > 17
> >> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> >> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> >> > > better.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours
are in
> >> the
> >> > > same
> >> > > > > > > > >>>>>>>>>>>>>>>>> file, we
> >> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to
tell it
> >> which
> >> > > one
> >> > > > > > > > >>>>>>>>>>>>>>>>> to
> >> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> >> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> >> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> >> > > > > > > > >>>>>>>>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level
=
> "L1";
> >> > > > lead_time
> >> > > > > > > > >>>>>>>>>>>>>>>>> =
> >> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> >> > > > > > > > >>>>>>>>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
> >> timestamps
> >> > are
> >> > > > off
> >> > > > > > > > >>>>>>>>>>>>>>>>> by 1
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour.
> >> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus
a 41
> hour
> >> > lead
> >> > > > > > > > >>>>>>>>>>>>>>>>> time
> >> > > > > > > > >>>>>>>>>>>>>>>>> =
> >> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> >> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a
24-hour
> >> max
> >> > > > value
> >> > > > > > > > >>>>>>>>>>>>>>>>> from
> >> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
valid at
> >> 05Z
> >> > > > while
> >> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> >> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
> >> Point-Stat
> >> > > > config
> >> > > > > > > > >>>>>>>>>>>>>>>>> file by
> >> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look
back 1
> >> hour:
> >> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> >> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM
hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files
and
> was
> >> > > fixed.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John
Halley
> Gotway
> >> > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the
time
> >> summaries,
> >> > > > but I
> >> > > > > > > > >>>>>>>>>>>>>>>>>> found
> >> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a
look on
> >> > dakota
> >> > > > > > > > >>>>>>>>>>>>>>>>>> here:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of
pb2nc to
> >> compute
> >> > > 24-
> >> > > > > > > > >>>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped
> >> the
> >> > > > output
> >> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands
> >> to
> >> > > count
> >> > > > > > > > >>>>>>>>>>>>>>>>>> up
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> >> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> >> > > pb2nc_output.txt
> >> > > > |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wc
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> >> > > pb2nc_output.txt
> >> > > > |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> >> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> >> > pb2nc_output.txt |
> >> > > > wc
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> >> > pb2nc_output.txt |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> sort
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return
114,538...
> >> meaning
> >> > > that
> >> > > > > > > > >>>>>>>>>>>>>>>>>> all
> >> > > > > > > > >>>>>>>>>>>>>>>>>> of
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
unique.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and
(4)
> returns
> >> > > 6,555
> >> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
> >> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations
> >> are
> >> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID
in
> that
> >> > file
> >> > > is
> >> > > > > > > > >>>>>>>>>>>>>>>>>> named
> >> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
> pb2nc_output.txt |
> >> > grep
> >> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180527_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180527_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
> >> repeated in
> >> > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>> last two
> >> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see
why these
> >> > > > duplicates
> >> > > > > > > > >>>>>>>>>>>>>>>>>> are
> >> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM
hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John
Halley
> Gotway
> >> > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's
no
> >> reason
> >> > why
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> have
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> >> > > across 2
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from
which
> we
> >> > could
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard
> >> Soh
> >> > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about
10
> minutes
> >> > > > (550.319
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
minutes.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will
not
> >> check
> >> > in
> >> > > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
files.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02
PM hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the
output (the
> >> > content
> >> > > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test
for time
> >> > > summary
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple
BURF
> >> > inputs.
> >> > > I
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two
input
> files
> >> and
> >> > > > will
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
Halley
> >> Gotway
> >> > > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this
will
> >> change
> >> > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this
case not
> >> > > currently
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I
add
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00
PM
> Howard
> >> > Soh
> >> > > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset
for time
> >> > > summary
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to
SVN.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
> >> /d3/personal/hsoh/data/pb2nc_summary/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52
PM
> John
> >> > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates
to the
> >> > logic
> >> > > > for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying
to
> >> compute
> >> > > > 24-hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see
these
> obs/hdr
> >> > > > counts:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(121832
> >> > currently)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
> Rscript
> >> to
> >> > > dump
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
> AIRNOW_MAX_240000
> >> > > string
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem
is in
> >> PB2NC
> >> > > not
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
summaries... or
> >> in
> >> > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take
another look
> in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
1:54 PM
> John
> >> > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging
into this.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
1:41 PM
> >> Howard
> >> > > > Soh <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291
observations
> from
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > and
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> >> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem
for the
> obs
> >> > > data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at
2:28 PM
> >> John
> >> > > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
Could
> you
> >> > > please
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> >> > PB2NC.
> >> > > > I
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc
PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
request
> >> > > 24-hourly
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z,
like
> >> this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
> -24*3600;
> >> > end =
> >> > > > 0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO"
];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max"
];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
> observations...
> >> > but
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which
is not
> >> what
> >> > > you'd
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The
hdr_typ_table
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> >> > > > "AIRNOW_MAX_240000"
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
> pointed
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
referenced.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> >> > caused
> >> > > by
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I
need 2
> >> of
> >> > > them
> >> > > > to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to
04Z
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a
look?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:44 PM
> >> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> >> > > PrepBUFR
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting
to
> >> compute a
> >> > 24
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more
together next
> >> > week.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:40
> PM
> >> > hsoh
> >> > > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R
script is
> >> > > reasonable
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
> observation
> >> > data
> >> > > in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
index of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the
NetCDF file
> >> was
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from
pb2nc
> >> > without
> >> > > > any
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it
has MORE
> >> > > headers
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
> 118193=headers).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables
from the
> >> > NetCDF
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(7295
> >> > > currently)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
value
> >> from 0
> >> > > to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message
type
> for
> >> > > > summary)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of
hdr_typ
> >> variable.
> >> > > The
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger
> >> than
> >> > > > 3655.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems
to be
> OK.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > >
> >> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEDIAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_RANGE_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_SDEV_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEDIAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_RANGE_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_SDEV_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM,
John
> >> Halley
> >> > > > Gotway
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a
look on
> >> > dakota
> >> > > > in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to
dump
> it
> >> to
> >> > > > ascii:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> >> > > > /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > >
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> >> > pb2nc_output.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error
but
> lists
> >> > all
> >> > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as
AIRNOW. But
> >> some
> >> > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000...
as
> >> evidenced
> >> > by
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v
hdr_typ_table
> >> > > > pb2nc_output.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000"
;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
indexing
> or
> >> > table
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Tue Jul 09 15:25:40 2019
I'm not sure that this has, indeed, been resolved. What's the latest?
Perry
On Tue, Jul 9, 2019 at 5:24 PM David Fillmore via RT
<met_help at ucar.edu>
wrote:
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Tue Jul 09 15:43:52 2019
Hi, John,
I've received two emails saying that this issue is resolved - but I
don't
remember hearing anything about a resolution. Did I miss it?
Thanks!
Perry
On Thu, Jun 6, 2019 at 6:21 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry (and Howard),
>
> I isolated this issue down and wrote up a GitHub issue describing
the
> problem:
> https://github.com/NCAR/MET/issues/1137
>
> Howard, please take a look and let me know if you need any help
further
> debugging this.
>
> Thanks,
> John
>
> On Thu, Jun 6, 2019 at 3:21 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Perry,
> >
> > I was able to replicate this behavior both on WCOSS and locally.
I
> > believe it's a bug in PB2NC. I'm trying to narrow it down and
will then
> > hand it off to Howard to debug. Thanks for identifying this
issue.
> >
> > John
> >
> > On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >>
> >> Hi, John,
> >>
> >> Well, it does appear that I got the same error with MET 8.1 as I
did
> with
> >> the previous version:
> >>
> >> ERROR :
> >> ERROR : StringArray::operator[](int) const -> range check error!
> >> ERROR :
> >>
> >> Perry
> >>
> >> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Perry,
> >> >
> >> > Did you ever get this issue resolved from 1 month ago? Or
should I
> log
> >> on
> >> > to WCOSS and take a look?
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Hi, John,
> >> > >
> >> > > Well, I am running the script in batch so I don't control
what node
> it
> >> > > gets. I can tell you that the working directory is here:
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > >
> >> > > And the output log file is here:
> >> > >
/ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688, which
> >> > should
> >> > > help you figure out how the thing is run.
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Perry
> >> > >
> >> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> >> > > met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Perry,
> >> > > >
> >> > > > We can log on to wcoss and test there instead. Yes, pb2nc
does
> >> seem to
> >> > > be
> >> > > > the culprit. Can you point us the wcoss node you're using
and the
> >> > > working
> >> > > > directory there? We basically need to recreate your call
to pb2nc
> >> to
> >> > try
> >> > > > to replicate the behavior.
> >> > > >
> >> > > > Thanks,
> >> > > > John
> >> > > >
> >> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via
RT <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >> > > > >
> >> > > > > Hi, David, John,
> >> > > > >
> >> > > > > I am not running on Theia. In fact I think it would take
quite
> a
> >> bit
> >> > > of
> >> > > > > effort to transition this thing to Theia, and I'll need
to be
> >> running
> >> > > > this
> >> > > > > verification on prod data anyway, which is only on WCOSS.
> >> > > > >
> >> > > > > However, if you wish to troubleshoot on Theia, you should
do so,
> >> and
> >> > I
> >> > > > > think you have all you need to be able to do so, right?
> >> > > > >
> >> > > > > It would seem that the error on my side is in PB2NC, so
I'll
> >> probably
> >> > > > need
> >> > > > > to make sure I'm running what you're running.
> >> > > > >
> >> > > > > Thanks!
> >> > > > >
> >> > > > > Perry
> >> > > > >
> >> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> >> > > met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Perry -
> >> > > > > > Are you running on theia?
> >> > > > > > John suggests that we try to troubleshoot there ...
> >> > > > > > thanks,
> >> > > > > > David
> >> > > > > >
> >> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> >> > > > > > > Hi Perry -
> >> > > > > > >
> >> > > > > > > An update on where we are on this issue:
> >> > > > > > > Using the prepda.2019042312.nc file you provided we
indeed
> >> > > reproduce
> >> > > > > > > the point_stat error.
> >> > > > > > > The point_stat error is due to only a single entry
the valid
> >> time
> >> > > > > > > table,
> >> > > > > > > e.g. from ncdump -v hdr_vld_table
> >> > > > > > > hdr_vld_table =
> >> > > > > > > "20190423_050000",
> >> > > > > > > while the hdr_vld indices in that file are hdr_vld =
0, 33,
> >> 33,
> >> > ...
> >> > > > > > > which are all out of range except the first.
> >> > > > > > >
> >> > > > > > > If I run pb2nc here with your PREPBUFR data and your
> >> > > PB2NCConfig_aqm
> >> > > > > > > config file, however,
> >> > > > > > > I generate a different prepda.2019042312.nc file that
runs
> >> > through
> >> > > > to
> >> > > > > > > completion with point_stat.
> >> > > > > > > Its valid time table:
> >> > > > > > >
> >> > > > > > > hdr_vld_table =
> >> > > > > > > "20190423_030000",
> >> > > > > > > "20190423_010000",
> >> > > > > > > "20190423_020000",
> >> > > > > > > "20190423_230000",
> >> > > > > > > "20190423_150000",
> >> > > > > > > "20190423_090000",
> >> > > > > > > "20190423_070000",
> >> > > > > > > "20190423_060000",
> >> > > > > > > "20190423_210000",
> >> > > > > > > "20190423_050000",
> >> > > > > > > "20190423_170000",
> >> > > > > > > "20190423_100000",
> >> > > > > > > "20190423_200000",
> >> > > > > > > "20190423_140000",
> >> > > > > > > "20190423_130000",
> >> > > > > > > "20190423_160000",
> >> > > > > > > "20190423_040000",
> >> > > > > > > "20190423_180000",
> >> > > > > > > "20190423_080000",
> >> > > > > > > "20190423_120000",
> >> > > > > > > "20190423_110000",
> >> > > > > > > "20190423_220000",
> >> > > > > > > "20190423_190000",
> >> > > > > > > "20190424_000000" ;
> >> > > > > > > }
> >> > > > > > >
> >> > > > > > > So it seems pb2nc is running correctly here, but not
on your
> >> > side.
> >> > > > > > > Let me discuss with John the best way to proceed ...
> >> > > > > > > thanks,
> >> > > > > > > David
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
wrote:
> >> > > > > > > > Hi, everyone,
> >> > > > > > > >
> >> > > > > > > > I am attaching this to this long thread regarding
the
> daily
> >> max
> >> > > of
> >> > > > > > > > ozone
> >> > > > > > > > verification, so we remember what we were doing
back then.
> >> > > > > > > >
> >> > > > > > > > I am now running this again using MET 8.0, and
something
> >> seems
> >> > to
> >> > > > not
> >> > > > > > > > be
> >> > > > > > > > working when I run point_stat. I get this error
when
> >> running a
> >> > > > daily
> >> > > > > > > > max
> >> > > > > > > > of ozone:
> >> > > > > > > >
> >> > > > > > > > ERROR :
> >> > > > > > > > ERROR : StringArray::operator[](int) const ->
range check
> >> > error!
> >> > > > > > > > ERROR :
> >> > > > > > > >
> >> > > > > > > > I am using my own scripts to run this using MET, so
this
> is
> >> not
> >> > > > using
> >> > > > > > > > METplus.
> >> > > > > > > >
> >> > > > > > > > You can find a log file of this here:
> >> > > > > > > >
> >> > > > >
> >> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> >> > > > > > > > It
> >> > > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc
code is
> >> > > working
> >> > > > > > > > fine, I
> >> > > > > > > > believe.
> >> > > > > > > >
> >> > > > > > > > The script I am using to run this is here:
> >> > > > > > > >
> >> >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> >> > > > > > > >
> >> > > > > > > > The prepbufr files are:
> >> > > > > > > >
> >> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
> and
> >> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> >> > > > > > > >
> >> > > > > > > > The model file is here:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > What could be causing this error?
> >> > > > > > > >
> >> > > > > > > > Thanks!
> >> > > > > > > >
> >> > > > > > > > Perry
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<
> >> > > > johnhg at ucar.edu>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Great, glad that did the trick.
> >> > > > > > > > >
> >> > > > > > > > > John
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran -
NOAA
> >> > Affiliate
> >> > > <
> >> > > > > > > > > perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Well this worked! And Jeff confirmed to me that
the
> 8-hr
> >> > > > average
> >> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z
like in
> >> the
> >> > > 1-hr
> >> > > > > > > > >> avg.
> >> > > > > > > > >>
> >> > > > > > > > >> Thanks!
> >> > > > > > > > >>
> >> > > > > > > > >> Perry
> >> > > > > > > > >>
> >> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley
Gotway
> >> > > > > > > > >> <johnhg at ucar.edu>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >>
> >> > > > > > > > >>> Perry,
> >> > > > > > > > >>>
> >> > > > > > > > >>> I think the problem is in your setting of
> >> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
> >> > > > > > > > >>>
> >> > > > > > > > >>> Here's your wgrib2 output:
> >> > > > > > > > >>> wgrib2 oz_DAY1
> >> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour
> ave
> >> > fcst:
> >> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-
1-22
> hour
> >> > ave
> >> > > > > > > > >>> fcst:
> >> > > > > > > > >>>
> >> > > > > > > > >>> The "d=2018081112" tells you the model
initialization
> >> time.
> >> > > > The
> >> > > > > > > > >>> valid
> >> > > > > > > > >>> time = init time + forecast time. So the
OZMAX1 data
> is
> >> > > valid
> >> > > > 16
> >> > > > > > > > >>> hours
> >> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
later.
> >> Rerun
> >> > > > that
> >> > > > > > > > >>> wgrib2
> >> > > > > > > > >>> command using the "-vt" argument to list the
> >> verification
> >> > > time.
> >> > > > > > > > >>>
> >> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data
because that
> >> data
> >> > > > isn't
> >> > > > > > > > >>> valid
> >> > > > > > > > >>> at the time you're requesting.
> >> > > > > > > > >>>
> >> > > > > > > > >>> John
> >> > > > > > > > >>>
> >> > > > > > > > >>>
> >> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran -
NOAA
> >> > > Affiliate
> >> > > > <
> >> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>
> >> > > > > > > > >>>> Hi, John,
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I haven't been able to work on this for
awhile, but
> now
> >> > I'm
> >> > > > > > > > >>>> working on
> >> > > > > > > > >>>> it again but seem to be running into issues.
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I have a file that has both the daily max for
1-hr
> >> ozone
> >> > and
> >> > > > the
> >> > > > > > > > >>>> daily
> >> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks
like
> this:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> 9 (t10a2)
> >> > > > > > > > >>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>> >
> >> > > > > > > > >>>> wgrib2 oz_DAY1
> >> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour ave
> >> fcst:
> >> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-
22 hour
> >> ave
> >> > > > fcst:
> >> > > > > > > > >>>> 10 (t10a2)
> >> > > > > > > > >>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>> >
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> As you can see they have different forecast
times.
> But
> >> > > when I
> >> > > > > > > > >>>> run
> >> > > > > > > > >>>> point_stat, I get this output:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>>
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array()
- No
> >> > matching
> >> > > > > > > > >>>> records
> >> > > > > > > > >>>> found for 'OZMAX8/L1'
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no
fields
> >> matching
> >> > > > > > > > >>>> OZMAX8/L1
> >> > > > > > > > >>>> found in file: oz_DAY1
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Here is the relevant portion of the point_stat
config
> >> > file:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> fcst = {
> >> > > > > > > > >>>> field = [
> >> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> >> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> >> > > > > > > > >>>> ];
> >> > > > > > > > >>>> }
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> obs = {
> >> > > > > > > > >>>> convert(x) = x * 10^9;
> >> > > > > > > > >>>> field = [
> >> > > > > > > > >>>> { name = "COPO"; level = "A1";
message_type =
> >> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
> >> > > > > > > > >>>> { name = "COPO"; level = "A8";
message_type =
> >> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
> >> > > > > > > > >>>> ];
> >> > > > > > > > >>>> }
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I don't remember if I had this working
previously for
> >> both
> >> > > the
> >> > > > > > > > >>>> 1hr
> >> > > > > > > > >>>> and
> >> > > > > > > > >>>> the 8hr ozone, but apparently there is some
sort of
> >> issue
> >> > > for
> >> > > > > > > > >>>> the
> >> > > > > > > > >>>> 8hr
> >> > > > > > > > >>>> ozone. Have any thoughts?
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Thanks!
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Perry
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>> <
> >> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>>> I chose option 2, and that seems to work - I
have a
> >> .stat
> >> > > > file
> >> > > > > > > > >>>>> with
> >> > > > > > > > >>>>> output. Now I'll need to do some work here
to see
> if
> >> the
> >> > > > > > > > >>>>> output
> >> > > > > > > > >>>>> looks OK,
> >> > > > > > > > >>>>> and to configure the verification so it
verifies
> >> OZMAX1
> >> > and
> >> > > > > > > > >>>>> OZMAX8.
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> I also wonder if this would be easily
modified for
> PM.
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> Perry
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> >> > > > > > > > >>>>> <johnhg at ucar.edu>
> >> > > > > > > > >>>>> wrote:
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>>> Perry,
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute
time
> >> > summaries
> >> > > > > > > > >>>>>> that
> >> > > > > > > > >>>>>> are
> >> > > > > > > > >>>>>> valid at 04Z. And you're comparing them to
> forecasts
> >> > that
> >> > > > are
> >> > > > > > > > >>>>>> purportedly
> >> > > > > > > > >>>>>> valid at 05Z. And that's the reason why you
have
> so
> >> > many
> >> > > > obs
> >> > > > > > > > >>>>>> rejected due
> >> > > > > > > > >>>>>> to the valid time.
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> It's up to you guys to decide what the
"correct"
> way
> >> of
> >> > > > > > > > >>>>>> handling
> >> > > > > > > > >>>>>> this
> >> > > > > > > > >>>>>> is... but here are a couple of options:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> (1) You could use the existing setup and
address
> this
> >> > > logic
> >> > > > in
> >> > > > > > > > >>>>>> Point-Stat config file. Change the matching
time
> >> > > obs_window
> >> > > > > > > > >>>>>> from 0 to:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> obs_window = {
> >> > > > > > > > >>>>>> beg = 3600;
> >> > > > > > > > >>>>>> end = 3600;
> >> > > > > > > > >>>>>> }
> >> > > > > > > > >>>>>> This tells Point-Stat to match observations
which
> are
> >> > > > exactly
> >> > > > > > > > >>>>>> 1-
> >> > > > > > > > >>>>>> hour
> >> > > > > > > > >>>>>> prior to the valid time of the forecast.
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary
> >> from
> >> > > 05Z
> >> > > > > > > > >>>>>> to
> >> > > > > > > > >>>>>> 05Z
> >> > > > > > > > >>>>>> instead and then leave the Point-Stat
obs_window
> >> setting
> >> > > at
> >> > > > 0:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> time_summary = {
> >> > > > > > > > >>>>>> flag = TRUE;
> >> > > > > > > > >>>>>> beg = "050000";
> >> > > > > > > > >>>>>> end = "050000";
> >> > > > > > > > >>>>>> ...
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> Thanks,
> >> > > > > > > > >>>>>> John
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry
Shafran -
> NOAA
> >> > > > > > > > >>>>>> Affiliate
> >> > > > > > > > >>>>>> <
> >> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
output
> >> file,
> >> > > but
> >> > > > > > > > >>>>>>> it's
> >> > > > > > > > >>>>>>> empty. Here's the output from point_stat:
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> DEBUG 4:
> >> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
> >> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> >> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> >> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> >> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> >> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> >> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> >> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1, for
> >> > > > observation
> >> > > > > > > > >>>>>>> type
> >> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
> >> interpolation
> >> > > > method
> >> > > > > > > > >>>>>>> NEAREST(1),
> >> > > > > > > > >>>>>>> using 0 pairs.
> >> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Looks like most rejections were due to
valid time.
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Have any thoughts about this one?
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Thanks!
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Perry
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry
Shafran -
> >> NOAA
> >> > > > > > > > >>>>>>> Affiliate
> >> > > > > > > > >>>>>>> <
> >> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>> Hi, John,
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Looks like the changing of the time stamp
from
> _04
> >> to
> >> > > _05
> >> > > > > > > > >>>>>>>> did
> >> > > > > > > > >>>>>>>> the
> >> > > > > > > > >>>>>>>> trick as point_stat ran without an issue
with
> that
> >> > > change.
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Perry
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John
Halley
> >> Gotway <
> >> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>>> Perry,
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
> requesting
> >> > the
> >> > > > > > > > >>>>>>>>> timestamp
> >> > > > > > > > >>>>>>>>> of the data from that file. I think
switching
> the
> >> > > > > > > > >>>>>>>>> verification time to
> >> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix
it.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on
the
> >> > > > initialization
> >> > > > > > > > >>>>>>>>> times
> >> > > > > > > > >>>>>>>>> and valid times for this data. Here's
some
> useful
> >> > > wgrib2
> >> > > > > > > > >>>>>>>>> and
> >> > > > > > > > >>>>>>>>> MET commands
> >> > > > > > > > >>>>>>>>> to illustrate.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time
(-t),
> >> > forecast
> >> > > > > > > > >>>>>>>>> time
> >> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it
is
> using
> >> a
> >> > > > sample
> >> > > > > > > > >>>>>>>>> data file you
> >> > > > > > > > >>>>>>>>> sent to me. But remember I'm still using
the
> >> "old"
> >> > > > wgrib2
> >> > > > > > > > >>>>>>>>> so
> >> > > > > > > > >>>>>>>>> I have that
> >> > > > > > > > >>>>>>>>> funky forecast hour:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> >> > > > > > > > >>>>>>>>>
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624
hour
> ave
> >> > > > > > > > >>>>>>>>> fcst:vt=2018070505
> >> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> >> > fcst:vt=2018070605
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a
higher
> >> > verbosity
> >> > > > > > > > >>>>>>>>> level
> >> > > > > > > > >>>>>>>>> (-v
> >> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info
of the
> >> data
> >> > it
> >> > > is
> >> > > > > > > > >>>>>>>>> reading:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> # Read the first record
> >> > > > > > > > >>>>>>>>> plot_data_plane
> >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> >> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> # Read the second record
> >> > > > > > > > >>>>>>>>> plot_data_plane
> >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> >> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Hope that helps.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>> John
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
Shafran -
> >> NOAA
> >> > > > > > > > >>>>>>>>> Affiliate <
> >> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Hi, John,
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> When I implemented all the changes (and
also
> set
> >> it
> >> > up
> >> > > > so
> >> > > > > > > > >>>>>>>>>> it's
> >> > > > > > > > >>>>>>>>>> reading the beta6 version), I used this
> command:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> >> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1
prepda.2018070912.nc
> >> > > > > > > > >>>>>>>>>>
> >> > > > > >
> >> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> >> > > > > > > > >>>>>>>>>> -v 4
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> But then I got this error:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> DEBUG 2:
> >> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING:
MetGrib2DataFile::data_plane_array() -
> >> No
> >> > > > > > > > >>>>>>>>>> matching
> >> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() ->
no
> fields
> >> > > > matching
> >> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> ERROR :
> >> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() ->
no
> >> requested
> >> > > > > > > > >>>>>>>>>> forecast
> >> > > > > > > > >>>>>>>>>> data
> >> > > > > > > > >>>>>>>>>> found! Exiting...
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the
wgrib2
> >> output
> >> > for
> >> > > > the
> >> > > > > > > > >>>>>>>>>> file
> >> > > > > > > > >>>>>>>>>> in this case reads:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> 14 (t14a2)
> >> > > > > > > > >>>>>>>>>>
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > >
> >> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> >> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-
7-16
> hour
> >> ave
> >> > > > fcst:
> >> > > > > > > > >>>>>>>>>> 15 (t14a2)
> >> > > > > > > > >>>>>>>>>>
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be
20180708_04? You
> >> had
> >> > > > > > > > >>>>>>>>>> suggested
> >> > > > > > > > >>>>>>>>>> that it should be the time stamp of the
2nd
> day,
> >> but
> >> > > > maybe
> >> > > > > > > > >>>>>>>>>> it needs to be
> >> > > > > > > > >>>>>>>>>> the time stamp of the first day?
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
> >> elsewhere.
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John
Halley
> >> Gotway <
> >> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> Perry,
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air
quality vx
> we
> >> > did,
> >> > > > we
> >> > > > > > > > >>>>>>>>>>> used
> >> > > > > > > > >>>>>>>>>>> observation files named:
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs
and
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> And those files don't exist in the July
data
> you
> >> > just
> >> > > > > > > > >>>>>>>>>>> sent.
> >> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names
like:
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> >> > aqm.t12z.anowpm.pb.tm00
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
> >> aqm.t12z.anowpm.pb.tm120
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually
verifying
> >> against
> >> > the
> >> > > > > > > > >>>>>>>>>>> BUFR
> >> > > > > > > > >>>>>>>>>>> or
> >> > > > > > > > >>>>>>>>>>> PREPBUFR.
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> Let me just tell you the key points for
the
> >> config
> >> > > file
> >> > > > > > > > >>>>>>>>>>> settings...
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to
pass 2
> >> > input
> >> > > > > > > > >>>>>>>>>>> files
> >> > > > > > > > >>>>>>>>>>> -
> >> > > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> >> > > > > > > > >>>>>>>>>>> pb2nc \
> >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00 \
> >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc
> \
> >> > > > > > > > >>>>>>>>>>> PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>> -pbfile
> >> hourly.20180704/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up
the
> >> > time_summary
> >> > > > > > > > >>>>>>>>>>> like
> >> > > > > > > > >>>>>>>>>>> this:
> >> > > > > > > > >>>>>>>>>>> time_summary = {
> >> > > > > > > > >>>>>>>>>>> flag = TRUE;
> >> > > > > > > > >>>>>>>>>>> beg = "040000";
> >> > > > > > > > >>>>>>>>>>> end = "040000";
> >> > > > > > > > >>>>>>>>>>> step = 24*3600;
> >> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end
= 0; }
> >> //
> >> > > > > > > > >>>>>>>>>>> previous
> >> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> >> > > > > > > > >>>>>>>>>>> grib_code = [];
> >> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> >> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> >> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect
obs every
> >> hour
> >> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
> >> summaries
> >> > > when
> >> > > > at
> >> > > > > > > > >>>>>>>>>>> least
> >> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values
at
> each
> >> > > station
> >> > > > > > > > >>>>>>>>>>> over the
> >> > > > > > > > >>>>>>>>>>> defined time period. They're
identified in
> the
> >> > > output
> >> > > > by
> >> > > > > > > > >>>>>>>>>>> the message type:
> >> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup
the
> "fcst"
> >> and
> >> > > > "obs"
> >> > > > > > > > >>>>>>>>>>> sections like this for the max of the
1-hour
> >> data.
> >> > > > > > > > >>>>>>>>>>> Instead
> >> > > > > > > > >>>>>>>>>>> of specifying
> >> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may
be more
> >> > > > convenient
> >> > > > > > > > >>>>>>>>>>> to specify the
> >> > > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify
> >> the
> >> > > > config
> >> > > > > > > > >>>>>>>>>>> file, I'd suggest
> >> > > > > > > > >>>>>>>>>>> using environment variables:
> >> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> fcst = {
> >> > > > > > > > >>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> >> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> >> > > > > > > > >>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>> obs = {
> >> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> >> > > > > > > > >>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
> >> message_type =
> >> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> >> > > > > > > > >>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
Halley
> >> Gotway
> >> > <
> >> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
> Shafran -
> >> > NOAA
> >> > > > > > > > >>>>>>>>>>>> Affiliate <
> >> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all
the bc
> >> grids
> >> > > and
> >> > > > > > > > >>>>>>>>>>>>> focus on
> >> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time
to
> copy
> >> > files
> >> > > > > > > > >>>>>>>>>>>>> over
> >> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
> >> > > > > > > > >>>>>>>>>>>>> let him work on the files he has for
now and
> >> if
> >> > he
> >> > > > runs
> >> > > > > > > > >>>>>>>>>>>>> into problems, I'll
> >> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
least
> the
> >> > 24-hr
> >> > > > > > > > >>>>>>>>>>>>> max/avg files).
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
> jeff.mcqueen <
> >> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
> >> grids...not
> >> > > for
> >> > > > > > > > >>>>>>>>>>>>>> bias
> >> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus
grids, To
> be
> >> > > safe,
> >> > > > > > > > >>>>>>>>>>>>>> you
> >> > > > > > > > >>>>>>>>>>>>>> may want to provide
> >> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran -
NOAA
> >> > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over
to
> Theia
> >> > for
> >> > > > John
> >> > > > > > > > >>>>>>>>>>>>>> to do
> >> > > > > > > > >>>>>>>>>>>>>> some testing some production files.
Do you
> >> > think
> >> > > I
> >> > > > > > > > >>>>>>>>>>>>>> should use parallels
> >> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops
files are
> OK
> >> for
> >> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> >> > > > > > > > >>>>>>>>>>>>>> for AQ?
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think
that
> the
> >> ops
> >> > > > files
> >> > > > > > > > >>>>>>>>>>>>>> should
> >> > > > > > > > >>>>>>>>>>>>>> be OK for now.
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
> jeff.mcqueen
> >> <
> >> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about
the
> >> > > parallels.
> >> > > > > > > > >>>>>>>>>>>>>>> We
> >> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily
max
> hour
> >> > > labels
> >> > > > > > > > >>>>>>>>>>>>>>> but
> >> > > > > > > > >>>>>>>>>>>>>>> management did not
> >> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19
CMAQ
> >> > upgrade.
> >> > > > > > > > >>>>>>>>>>>>>>> Jeff
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels
have the
> >> > correct
> >> > > > time
> >> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if
I send
> >> John
> >> > > > over
> >> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> >> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has
any
> >> other
> >> > > > > > > > >>>>>>>>>>>>>>> versions
> >> > > > > > > > >>>>>>>>>>>>>>> of
> >> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already
have -
> but
> >> > they
> >> > > > can
> >> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
> >> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new
> >> does
> >> > > over
> >> > > > > > > > >>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
> >> jeff.mcqueen <
> >> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new,
you may
> >> want
> >> > to
> >> > > > use
> >> > > > > > > > >>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output
(para5).
> >> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> >> > > Jianping
> >> > > > > > > > >>>>>>>>>>>>>>>> also
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it
but
> >> > > management
> >> > > > > > > > >>>>>>>>>>>>>>>> did
> >> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> >> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to
do so
> >> with
> >> > the
> >> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
> >> > > > > > > > >>>>>>>>>>>>>>>> implementation.
> >> > > > > > > > >>>>>>>>>>>>>>>> Jeff
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
> >> conversation.
> >> > I
> >> > > > > > > > >>>>>>>>>>>>>>>> think
> >> > > > > > > > >>>>>>>>>>>>>>>> I
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the
timestamps
> some
> >> > time
> >> > > > ago,
> >> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or
with
> >> Jeff. I
> >> > > > think
> >> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> >> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not
too
> >> > entirely
> >> > > > > > > > >>>>>>>>>>>>>>>> sure.
> >> > > > > > > > >>>>>>>>>>>>>>>> But he would have
> >> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct
and
> how
> >> to
> >> > > > > > > > >>>>>>>>>>>>>>>> approach
> >> > > > > > > > >>>>>>>>>>>>>>>> this.
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time
stamp
> >> issues
> >> > > John
> >> > > > > > > > >>>>>>>>>>>>>>>> talks
> >> > > > > > > > >>>>>>>>>>>>>>>> about?
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM,
John
> Halley
> >> > > > Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all
squared
> away
> >> to
> >> > > > > > > > >>>>>>>>>>>>>>>>> compute
> >> > > > > > > > >>>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average
values for
> 1-
> >> and
> >> > > 8-
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >> > \
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like
this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> >> > > > > \
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
PointStatConfig_AIRNOW
> \
> >> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> >> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1
versus
> >> COPO/A1,
> >> > > for
> >> > > > > > > > >>>>>>>>>>>>>>>>> observation type
AIRNOW_MAX_240000, over
> >> > region
> >> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
> >> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> >> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147
pairs.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> >> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes
the
> >> contents
> >> > of
> >> > > > > > > > >>>>>>>>>>>>>>>>> this
> >> > > > > > > > >>>>>>>>>>>>>>>>> "max"
> >> > > > > > > > >>>>>>>>>>>>>>>>> file:
> >> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> >> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618
hour ave
> >> fcst:
> >> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
sigma
> >> > > > level:17-40
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty
funny. But
> >> > > testing
> >> > > > > > > > >>>>>>>>>>>>>>>>> revealed
> >> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
> forecast
> >> > > hours
> >> > > > 17
> >> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> >> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> >> > > better.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours
are in
> >> the
> >> > > same
> >> > > > > > > > >>>>>>>>>>>>>>>>> file, we
> >> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to
tell it
> >> which
> >> > > one
> >> > > > > > > > >>>>>>>>>>>>>>>>> to
> >> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> >> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> >> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> >> > > > > > > > >>>>>>>>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level
=
> "L1";
> >> > > > lead_time
> >> > > > > > > > >>>>>>>>>>>>>>>>> =
> >> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> >> > > > > > > > >>>>>>>>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
> >> timestamps
> >> > are
> >> > > > off
> >> > > > > > > > >>>>>>>>>>>>>>>>> by 1
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour.
> >> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus
a 41
> hour
> >> > lead
> >> > > > > > > > >>>>>>>>>>>>>>>>> time
> >> > > > > > > > >>>>>>>>>>>>>>>>> =
> >> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> >> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a
24-hour
> >> max
> >> > > > value
> >> > > > > > > > >>>>>>>>>>>>>>>>> from
> >> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
valid at
> >> 05Z
> >> > > > while
> >> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> >> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
> >> Point-Stat
> >> > > > config
> >> > > > > > > > >>>>>>>>>>>>>>>>> file by
> >> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look
back 1
> >> hour:
> >> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> >> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM
hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files
and
> was
> >> > > fixed.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John
Halley
> Gotway
> >> > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the
time
> >> summaries,
> >> > > > but I
> >> > > > > > > > >>>>>>>>>>>>>>>>>> found
> >> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a
look on
> >> > dakota
> >> > > > > > > > >>>>>>>>>>>>>>>>>> here:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of
pb2nc to
> >> compute
> >> > > 24-
> >> > > > > > > > >>>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped
> >> the
> >> > > > output
> >> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands
> >> to
> >> > > count
> >> > > > > > > > >>>>>>>>>>>>>>>>>> up
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> >> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> >> > > pb2nc_output.txt
> >> > > > |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wc
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> >> > > pb2nc_output.txt
> >> > > > |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> >> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> >> > pb2nc_output.txt |
> >> > > > wc
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> >> > pb2nc_output.txt |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> sort
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return
114,538...
> >> meaning
> >> > > that
> >> > > > > > > > >>>>>>>>>>>>>>>>>> all
> >> > > > > > > > >>>>>>>>>>>>>>>>>> of
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
unique.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and
(4)
> returns
> >> > > 6,555
> >> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
> >> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations
> >> are
> >> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID
in
> that
> >> > file
> >> > > is
> >> > > > > > > > >>>>>>>>>>>>>>>>>> named
> >> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
> pb2nc_output.txt |
> >> > grep
> >> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180527_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180527_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
> >> repeated in
> >> > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>> last two
> >> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see
why these
> >> > > > duplicates
> >> > > > > > > > >>>>>>>>>>>>>>>>>> are
> >> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM
hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John
Halley
> Gotway
> >> > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's
no
> >> reason
> >> > why
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> have
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> >> > > across 2
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from
which
> we
> >> > could
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard
> >> Soh
> >> > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about
10
> minutes
> >> > > > (550.319
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
minutes.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will
not
> >> check
> >> > in
> >> > > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
files.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02
PM hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the
output (the
> >> > content
> >> > > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test
for time
> >> > > summary
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple
BURF
> >> > inputs.
> >> > > I
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two
input
> files
> >> and
> >> > > > will
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
Halley
> >> Gotway
> >> > > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this
will
> >> change
> >> > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this
case not
> >> > > currently
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I
add
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00
PM
> Howard
> >> > Soh
> >> > > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset
for time
> >> > > summary
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to
SVN.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
> >> /d3/personal/hsoh/data/pb2nc_summary/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52
PM
> John
> >> > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates
to the
> >> > logic
> >> > > > for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying
to
> >> compute
> >> > > > 24-hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see
these
> obs/hdr
> >> > > > counts:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(121832
> >> > currently)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
> Rscript
> >> to
> >> > > dump
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
> AIRNOW_MAX_240000
> >> > > string
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem
is in
> >> PB2NC
> >> > > not
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
summaries... or
> >> in
> >> > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take
another look
> in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
1:54 PM
> John
> >> > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging
into this.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
1:41 PM
> >> Howard
> >> > > > Soh <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291
observations
> from
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > and
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> >> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem
for the
> obs
> >> > > data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at
2:28 PM
> >> John
> >> > > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
Could
> you
> >> > > please
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> >> > PB2NC.
> >> > > > I
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc
PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
request
> >> > > 24-hourly
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z,
like
> >> this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
> -24*3600;
> >> > end =
> >> > > > 0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO"
];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max"
];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
> observations...
> >> > but
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which
is not
> >> what
> >> > > you'd
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The
hdr_typ_table
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> >> > > > "AIRNOW_MAX_240000"
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
> pointed
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
referenced.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> >> > caused
> >> > > by
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I
need 2
> >> of
> >> > > them
> >> > > > to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to
04Z
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a
look?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:44 PM
> >> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> >> > > PrepBUFR
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting
to
> >> compute a
> >> > 24
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more
together next
> >> > week.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:40
> PM
> >> > hsoh
> >> > > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R
script is
> >> > > reasonable
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
> observation
> >> > data
> >> > > in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
index of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the
NetCDF file
> >> was
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from
pb2nc
> >> > without
> >> > > > any
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it
has MORE
> >> > > headers
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
> 118193=headers).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables
from the
> >> > NetCDF
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(7295
> >> > > currently)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
value
> >> from 0
> >> > > to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message
type
> for
> >> > > > summary)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of
hdr_typ
> >> variable.
> >> > > The
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger
> >> than
> >> > > > 3655.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems
to be
> OK.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > >
> >> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEDIAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_RANGE_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_SDEV_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEDIAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_RANGE_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_SDEV_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM,
John
> >> Halley
> >> > > > Gotway
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a
look on
> >> > dakota
> >> > > > in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to
dump
> it
> >> to
> >> > > > ascii:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> >> > > > /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > >
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> >> > pb2nc_output.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error
but
> lists
> >> > all
> >> > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as
AIRNOW. But
> >> some
> >> > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000...
as
> >> evidenced
> >> > by
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v
hdr_typ_table
> >> > > > pb2nc_output.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000"
;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
indexing
> or
> >> > table
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Howard Soh
Time: Tue Jul 09 15:57:11 2019
This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld branch).
It's not available yet for you. Julie will install it over the next
couple of days.
Cheers,
Howard
On Tue Jul 09 15:43:52 2019, perry.shafran at noaa.gov wrote:
> Hi, John,
>
> I've received two emails saying that this issue is resolved - but I
> don't
> remember hearing anything about a resolution. Did I miss it?
>
> Thanks!
>
> Perry
>
> On Thu, Jun 6, 2019 at 6:21 PM John Halley Gotway via RT
> <met_help at ucar.edu>
> wrote:
>
> > Perry (and Howard),
> >
> > I isolated this issue down and wrote up a GitHub issue describing
the
> > problem:
> > https://github.com/NCAR/MET/issues/1137
> >
> > Howard, please take a look and let me know if you need any help
> > further
> > debugging this.
> >
> > Thanks,
> > John
> >
> > On Thu, Jun 6, 2019 at 3:21 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Perry,
> > >
> > > I was able to replicate this behavior both on WCOSS and locally.
I
> > > believe it's a bug in PB2NC. I'm trying to narrow it down and
will
> > > then
> > > hand it off to Howard to debug. Thanks for identifying this
issue.
> > >
> > > John
> > >
> > > On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >>
> > >> Hi, John,
> > >>
> > >> Well, it does appear that I got the same error with MET 8.1 as
I
> > >> did
> > with
> > >> the previous version:
> > >>
> > >> ERROR :
> > >> ERROR : StringArray::operator[](int) const -> range check
error!
> > >> ERROR :
> > >>
> > >> Perry
> > >>
> > >> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> > Perry,
> > >> >
> > >> > Did you ever get this issue resolved from 1 month ago? Or
> > >> > should I
> > log
> > >> on
> > >> > to WCOSS and take a look?
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT
<
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > > Hi, John,
> > >> > >
> > >> > > Well, I am running the script in batch so I don't control
what
> > >> > > node
> > it
> > >> > > gets. I can tell you that the working directory is here:
> > >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >> > >
> > >> > > And the output log file is here:
> > >> > >
/ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
> > >> > > which
> > >> > should
> > >> > > help you figure out how the thing is run.
> > >> > >
> > >> > > Thanks!
> > >> > >
> > >> > > Perry
> > >> > >
> > >> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> > >> > > met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > > > Perry,
> > >> > > >
> > >> > > > We can log on to wcoss and test there instead. Yes,
pb2nc
> > >> > > > does
> > >> seem to
> > >> > > be
> > >> > > > the culprit. Can you point us the wcoss node you're
using
> > >> > > > and the
> > >> > > working
> > >> > > > directory there? We basically need to recreate your call
to
> > >> > > > pb2nc
> > >> to
> > >> > try
> > >> > > > to replicate the behavior.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > John
> > >> > > >
> > >> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via
RT
> > >> > > > <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > >
> > >> > > > > <URL:
> > >> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > >> > > > >
> > >> > > > > Hi, David, John,
> > >> > > > >
> > >> > > > > I am not running on Theia. In fact I think it would
take
> > >> > > > > quite
> > a
> > >> bit
> > >> > > of
> > >> > > > > effort to transition this thing to Theia, and I'll need
to
> > >> > > > > be
> > >> running
> > >> > > > this
> > >> > > > > verification on prod data anyway, which is only on
WCOSS.
> > >> > > > >
> > >> > > > > However, if you wish to troubleshoot on Theia, you
should
> > >> > > > > do so,
> > >> and
> > >> > I
> > >> > > > > think you have all you need to be able to do so, right?
> > >> > > > >
> > >> > > > > It would seem that the error on my side is in PB2NC, so
> > >> > > > > I'll
> > >> probably
> > >> > > > need
> > >> > > > > to make sure I'm running what you're running.
> > >> > > > >
> > >> > > > > Thanks!
> > >> > > > >
> > >> > > > > Perry
> > >> > > > >
> > >> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> > >> > > met_help at ucar.edu>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi Perry -
> > >> > > > > > Are you running on theia?
> > >> > > > > > John suggests that we try to troubleshoot there ...
> > >> > > > > > thanks,
> > >> > > > > > David
> > >> > > > > >
> > >> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > >> > > > > > > Hi Perry -
> > >> > > > > > >
> > >> > > > > > > An update on where we are on this issue:
> > >> > > > > > > Using the prepda.2019042312.nc file you provided we
> > >> > > > > > > indeed
> > >> > > reproduce
> > >> > > > > > > the point_stat error.
> > >> > > > > > > The point_stat error is due to only a single entry
the
> > >> > > > > > > valid
> > >> time
> > >> > > > > > > table,
> > >> > > > > > > e.g. from ncdump -v hdr_vld_table
> > >> > > > > > > hdr_vld_table =
> > >> > > > > > > "20190423_050000",
> > >> > > > > > > while the hdr_vld indices in that file are hdr_vld
=
> > >> > > > > > > 0, 33,
> > >> 33,
> > >> > ...
> > >> > > > > > > which are all out of range except the first.
> > >> > > > > > >
> > >> > > > > > > If I run pb2nc here with your PREPBUFR data and
your
> > >> > > PB2NCConfig_aqm
> > >> > > > > > > config file, however,
> > >> > > > > > > I generate a different prepda.2019042312.nc file
that
> > >> > > > > > > runs
> > >> > through
> > >> > > > to
> > >> > > > > > > completion with point_stat.
> > >> > > > > > > Its valid time table:
> > >> > > > > > >
> > >> > > > > > > hdr_vld_table =
> > >> > > > > > > "20190423_030000",
> > >> > > > > > > "20190423_010000",
> > >> > > > > > > "20190423_020000",
> > >> > > > > > > "20190423_230000",
> > >> > > > > > > "20190423_150000",
> > >> > > > > > > "20190423_090000",
> > >> > > > > > > "20190423_070000",
> > >> > > > > > > "20190423_060000",
> > >> > > > > > > "20190423_210000",
> > >> > > > > > > "20190423_050000",
> > >> > > > > > > "20190423_170000",
> > >> > > > > > > "20190423_100000",
> > >> > > > > > > "20190423_200000",
> > >> > > > > > > "20190423_140000",
> > >> > > > > > > "20190423_130000",
> > >> > > > > > > "20190423_160000",
> > >> > > > > > > "20190423_040000",
> > >> > > > > > > "20190423_180000",
> > >> > > > > > > "20190423_080000",
> > >> > > > > > > "20190423_120000",
> > >> > > > > > > "20190423_110000",
> > >> > > > > > > "20190423_220000",
> > >> > > > > > > "20190423_190000",
> > >> > > > > > > "20190424_000000" ;
> > >> > > > > > > }
> > >> > > > > > >
> > >> > > > > > > So it seems pb2nc is running correctly here, but
not
> > >> > > > > > > on your
> > >> > side.
> > >> > > > > > > Let me discuss with John the best way to proceed
...
> > >> > > > > > > thanks,
> > >> > > > > > > David
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
> > >> > > > > > > wrote:
> > >> > > > > > > > Hi, everyone,
> > >> > > > > > > >
> > >> > > > > > > > I am attaching this to this long thread regarding
> > >> > > > > > > > the
> > daily
> > >> max
> > >> > > of
> > >> > > > > > > > ozone
> > >> > > > > > > > verification, so we remember what we were doing
back
> > >> > > > > > > > then.
> > >> > > > > > > >
> > >> > > > > > > > I am now running this again using MET 8.0, and
> > >> > > > > > > > something
> > >> seems
> > >> > to
> > >> > > > not
> > >> > > > > > > > be
> > >> > > > > > > > working when I run point_stat. I get this error
> > >> > > > > > > > when
> > >> running a
> > >> > > > daily
> > >> > > > > > > > max
> > >> > > > > > > > of ozone:
> > >> > > > > > > >
> > >> > > > > > > > ERROR :
> > >> > > > > > > > ERROR : StringArray::operator[](int) const ->
range
> > >> > > > > > > > check
> > >> > error!
> > >> > > > > > > > ERROR :
> > >> > > > > > > >
> > >> > > > > > > > I am using my own scripts to run this using MET,
so
> > >> > > > > > > > this
> > is
> > >> not
> > >> > > > using
> > >> > > > > > > > METplus.
> > >> > > > > > > >
> > >> > > > > > > > You can find a log file of this here:
> > >> > > > > > > >
> > >> > > > >
> > >> >
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > >> > > > > > > > It
> > >> > > > > > > > runs pb2nc and point_stat for max ozone. The
pb2nc
> > >> > > > > > > > code is
> > >> > > working
> > >> > > > > > > > fine, I
> > >> > > > > > > > believe.
> > >> > > > > > > >
> > >> > > > > > > > The script I am using to run this is here:
> > >> > > > > > > >
> > >> >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > >> > > > > > > >
> > >> > > > > > > > The prepbufr files are:
> > >> > > > > > > >
> > >> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
> > and
> > >> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > >> > > > > > > >
> > >> > > > > > > > The model file is here:
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > What could be causing this error?
> > >> > > > > > > >
> > >> > > > > > > > Thanks!
> > >> > > > > > > >
> > >> > > > > > > > Perry
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley
Gotway <
> > >> > > > johnhg at ucar.edu>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Great, glad that did the trick.
> > >> > > > > > > > >
> > >> > > > > > > > > John
> > >> > > > > > > > >
> > >> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran -
> > >> > > > > > > > > NOAA
> > >> > Affiliate
> > >> > > <
> > >> > > > > > > > > perry.shafran at noaa.gov> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Well this worked! And Jeff confirmed to me
that
> > >> > > > > > > > >> the
> > 8-hr
> > >> > > > average
> > >> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z
> > >> > > > > > > > >> like in
> > >> the
> > >> > > 1-hr
> > >> > > > > > > > >> avg.
> > >> > > > > > > > >>
> > >> > > > > > > > >> Thanks!
> > >> > > > > > > > >>
> > >> > > > > > > > >> Perry
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley
> > >> > > > > > > > >> Gotway
> > >> > > > > > > > >> <johnhg at ucar.edu>
> > >> > > > > > > > >> wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >>> Perry,
> > >> > > > > > > > >>>
> > >> > > > > > > > >>> I think the problem is in your setting of
> > >> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
> > >> > > > > > > > >>>
> > >> > > > > > > > >>> Here's your wgrib2 output:
> > >> > > > > > > > >>> wgrib2 oz_DAY1
> > >> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-
16
> > >> > > > > > > > >>> hour
> > ave
> > >> > fcst:
> > >> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-
1-22
> > hour
> > >> > ave
> > >> > > > > > > > >>> fcst:
> > >> > > > > > > > >>>
> > >> > > > > > > > >>> The "d=2018081112" tells you the model
> > >> > > > > > > > >>> initialization
> > >> time.
> > >> > > > The
> > >> > > > > > > > >>> valid
> > >> > > > > > > > >>> time = init time + forecast time. So the
OZMAX1
> > >> > > > > > > > >>> data
> > is
> > >> > > valid
> > >> > > > 16
> > >> > > > > > > > >>> hours
> > >> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
> > >> > > > > > > > >>> later.
> > >> Rerun
> > >> > > > that
> > >> > > > > > > > >>> wgrib2
> > >> > > > > > > > >>> command using the "-vt" argument to list the
> > >> verification
> > >> > > time.
> > >> > > > > > > > >>>
> > >> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data
because
> > >> > > > > > > > >>> that
> > >> data
> > >> > > > isn't
> > >> > > > > > > > >>> valid
> > >> > > > > > > > >>> at the time you're requesting.
> > >> > > > > > > > >>>
> > >> > > > > > > > >>> John
> > >> > > > > > > > >>>
> > >> > > > > > > > >>>
> > >> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran
-
> > >> > > > > > > > >>> NOAA
> > >> > > Affiliate
> > >> > > > <
> > >> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
> > >> > > > > > > > >>>
> > >> > > > > > > > >>>> Hi, John,
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> I haven't been able to work on this for
awhile,
> > >> > > > > > > > >>>> but
> > now
> > >> > I'm
> > >> > > > > > > > >>>> working on
> > >> > > > > > > > >>>> it again but seem to be running into issues.
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> I have a file that has both the daily max
for
> > >> > > > > > > > >>>> 1-hr
> > >> ozone
> > >> > and
> > >> > > > the
> > >> > > > > > > > >>>> daily
> > >> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks
> > >> > > > > > > > >>>> like
> > this:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> 9 (t10a2)
> > >> > > > > > > > >>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >> > > > > > > > >>>> >
> > >> > > > > > > > >>>> wgrib2 oz_DAY1
> > >> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
> > >> > > > > > > > >>>> hour ave
> > >> fcst:
> > >> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-
1-
> > >> > > > > > > > >>>> 22 hour
> > >> ave
> > >> > > > fcst:
> > >> > > > > > > > >>>> 10 (t10a2)
> > >> > > > > > > > >>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >> > > > > > > > >>>> >
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> As you can see they have different forecast
> > >> > > > > > > > >>>> times.
> > But
> > >> > > when I
> > >> > > > > > > > >>>> run
> > >> > > > > > > > >>>> point_stat, I get this output:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> DEBUG 2:
> > >> > > > > > > > >>>> DEBUG 2:
> > >> > > > > > > > >>>>
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
--------------------------------------------------------------------------------
> > >> > > > > > > > >>>> DEBUG 2:
> > >> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > >> > > > > > > > >>>> WARNING:
> > >> > > > > > > > >>>> WARNING:
MetGrib2DataFile::data_plane_array() -
> > >> > > > > > > > >>>> No
> > >> > matching
> > >> > > > > > > > >>>> records
> > >> > > > > > > > >>>> found for 'OZMAX8/L1'
> > >> > > > > > > > >>>> WARNING:
> > >> > > > > > > > >>>> WARNING:
> > >> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no
> > >> > > > > > > > >>>> fields
> > >> matching
> > >> > > > > > > > >>>> OZMAX8/L1
> > >> > > > > > > > >>>> found in file: oz_DAY1
> > >> > > > > > > > >>>> WARNING:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> Here is the relevant portion of the
point_stat
> > >> > > > > > > > >>>> config
> > >> > file:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> fcst = {
> > >> > > > > > > > >>>> field = [
> > >> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > >> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > >> > > > > > > > >>>> ];
> > >> > > > > > > > >>>> }
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> obs = {
> > >> > > > > > > > >>>> convert(x) = x * 10^9;
> > >> > > > > > > > >>>> field = [
> > >> > > > > > > > >>>> { name = "COPO"; level = "A1";
> > >> > > > > > > > >>>> message_type =
> > >> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
> > >> > > > > > > > >>>> { name = "COPO"; level = "A8";
> > >> > > > > > > > >>>> message_type =
> > >> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
> > >> > > > > > > > >>>> ];
> > >> > > > > > > > >>>> }
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> I don't remember if I had this working
> > >> > > > > > > > >>>> previously for
> > >> both
> > >> > > the
> > >> > > > > > > > >>>> 1hr
> > >> > > > > > > > >>>> and
> > >> > > > > > > > >>>> the 8hr ozone, but apparently there is some
> > >> > > > > > > > >>>> sort of
> > >> issue
> > >> > > for
> > >> > > > > > > > >>>> the
> > >> > > > > > > > >>>> 8hr
> > >> > > > > > > > >>>> ozone. Have any thoughts?
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> Thanks!
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> Perry
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry
Shafran
> > >> > > > > > > > >>>> - NOAA
> > >> > > > Affiliate
> > >> > > > > > > > >>>> <
> > >> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > >> > > > > > > > >>>>
> > >> > > > > > > > >>>>> I chose option 2, and that seems to work -
I
> > >> > > > > > > > >>>>> have a
> > >> .stat
> > >> > > > file
> > >> > > > > > > > >>>>> with
> > >> > > > > > > > >>>>> output. Now I'll need to do some work here
to
> > >> > > > > > > > >>>>> see
> > if
> > >> the
> > >> > > > > > > > >>>>> output
> > >> > > > > > > > >>>>> looks OK,
> > >> > > > > > > > >>>>> and to configure the verification so it
> > >> > > > > > > > >>>>> verifies
> > >> OZMAX1
> > >> > and
> > >> > > > > > > > >>>>> OZMAX8.
> > >> > > > > > > > >>>>>
> > >> > > > > > > > >>>>> I also wonder if this would be easily
modified
> > >> > > > > > > > >>>>> for
> > PM.
> > >> > > > > > > > >>>>>
> > >> > > > > > > > >>>>> Perry
> > >> > > > > > > > >>>>>
> > >> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John
Halley
> > >> > > > > > > > >>>>> Gotway
> > >> > > > > > > > >>>>> <johnhg at ucar.edu>
> > >> > > > > > > > >>>>> wrote:
> > >> > > > > > > > >>>>>
> > >> > > > > > > > >>>>>> Perry,
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to
compute
> > >> > > > > > > > >>>>>> time
> > >> > summaries
> > >> > > > > > > > >>>>>> that
> > >> > > > > > > > >>>>>> are
> > >> > > > > > > > >>>>>> valid at 04Z. And you're comparing them
to
> > forecasts
> > >> > that
> > >> > > > are
> > >> > > > > > > > >>>>>> purportedly
> > >> > > > > > > > >>>>>> valid at 05Z. And that's the reason why
you
> > >> > > > > > > > >>>>>> have
> > so
> > >> > many
> > >> > > > obs
> > >> > > > > > > > >>>>>> rejected due
> > >> > > > > > > > >>>>>> to the valid time.
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> It's up to you guys to decide what the
> > >> > > > > > > > >>>>>> "correct"
> > way
> > >> of
> > >> > > > > > > > >>>>>> handling
> > >> > > > > > > > >>>>>> this
> > >> > > > > > > > >>>>>> is... but here are a couple of options:
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> (1) You could use the existing setup and
> > >> > > > > > > > >>>>>> address
> > this
> > >> > > logic
> > >> > > > in
> > >> > > > > > > > >>>>>> Point-Stat config file. Change the
matching
> > >> > > > > > > > >>>>>> time
> > >> > > obs_window
> > >> > > > > > > > >>>>>> from 0 to:
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> obs_window = {
> > >> > > > > > > > >>>>>> beg = 3600;
> > >> > > > > > > > >>>>>> end = 3600;
> > >> > > > > > > > >>>>>> }
> > >> > > > > > > > >>>>>> This tells Point-Stat to match
observations
> > >> > > > > > > > >>>>>> which
> > are
> > >> > > > exactly
> > >> > > > > > > > >>>>>> 1-
> > >> > > > > > > > >>>>>> hour
> > >> > > > > > > > >>>>>> prior to the valid time of the forecast.
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a
time
> > >> > > > > > > > >>>>>> summary
> > >> from
> > >> > > 05Z
> > >> > > > > > > > >>>>>> to
> > >> > > > > > > > >>>>>> 05Z
> > >> > > > > > > > >>>>>> instead and then leave the Point-Stat
> > >> > > > > > > > >>>>>> obs_window
> > >> setting
> > >> > > at
> > >> > > > 0:
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> time_summary = {
> > >> > > > > > > > >>>>>> flag = TRUE;
> > >> > > > > > > > >>>>>> beg = "050000";
> > >> > > > > > > > >>>>>> end = "050000";
> > >> > > > > > > > >>>>>> ...
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> Thanks,
> > >> > > > > > > > >>>>>> John
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry
> > >> > > > > > > > >>>>>> Shafran -
> > NOAA
> > >> > > > > > > > >>>>>> Affiliate
> > >> > > > > > > > >>>>>> <
> > >> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > >> > > > > > > > >>>>>>
> > >> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
> > >> > > > > > > > >>>>>>> output
> > >> file,
> > >> > > but
> > >> > > > > > > > >>>>>>> it's
> > >> > > > > > > > >>>>>>> empty. Here's the output from
point_stat:
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>> DEBUG 4:
> > >> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
> > >> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > >> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > >> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > >> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > >> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > >> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > >> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
> > >> > > > > > > > >>>>>>> COPO/A1, for
> > >> > > > observation
> > >> > > > > > > > >>>>>>> type
> > >> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
> > >> interpolation
> > >> > > > method
> > >> > > > > > > > >>>>>>> NEAREST(1),
> > >> > > > > > > > >>>>>>> using 0 pairs.
> > >> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > >> > > > > > > > >>>>>>> DEBUG 3: Observations processed =
124637
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time =
122154
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > >> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>> Looks like most rejections were due to
valid
> > >> > > > > > > > >>>>>>> time.
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>> Have any thoughts about this one?
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>> Thanks!
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>> Perry
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry
> > >> > > > > > > > >>>>>>> Shafran -
> > >> NOAA
> > >> > > > > > > > >>>>>>> Affiliate
> > >> > > > > > > > >>>>>>> <
> > >> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > >> > > > > > > > >>>>>>>
> > >> > > > > > > > >>>>>>>> Hi, John,
> > >> > > > > > > > >>>>>>>>
> > >> > > > > > > > >>>>>>>> Looks like the changing of the time
stamp
> > >> > > > > > > > >>>>>>>> from
> > _04
> > >> to
> > >> > > _05
> > >> > > > > > > > >>>>>>>> did
> > >> > > > > > > > >>>>>>>> the
> > >> > > > > > > > >>>>>>>> trick as point_stat ran without an issue
> > >> > > > > > > > >>>>>>>> with
> > that
> > >> > > change.
> > >> > > > > > > > >>>>>>>>
> > >> > > > > > > > >>>>>>>> Thanks!
> > >> > > > > > > > >>>>>>>>
> > >> > > > > > > > >>>>>>>> Perry
> > >> > > > > > > > >>>>>>>>
> > >> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John
> > >> > > > > > > > >>>>>>>> Halley
> > >> Gotway <
> > >> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>
> > >> > > > > > > > >>>>>>>>> Perry,
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
> > requesting
> > >> > the
> > >> > > > > > > > >>>>>>>>> timestamp
> > >> > > > > > > > >>>>>>>>> of the data from that file. I think
> > >> > > > > > > > >>>>>>>>> switching
> > the
> > >> > > > > > > > >>>>>>>>> verification time to
> > >> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would
fix
> > >> > > > > > > > >>>>>>>>> it.
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on
> > >> > > > > > > > >>>>>>>>> the
> > >> > > > initialization
> > >> > > > > > > > >>>>>>>>> times
> > >> > > > > > > > >>>>>>>>> and valid times for this data. Here's
> > >> > > > > > > > >>>>>>>>> some
> > useful
> > >> > > wgrib2
> > >> > > > > > > > >>>>>>>>> and
> > >> > > > > > > > >>>>>>>>> MET commands
> > >> > > > > > > > >>>>>>>>> to illustrate.
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init
time
> > >> > > > > > > > >>>>>>>>> (-t),
> > >> > forecast
> > >> > > > > > > > >>>>>>>>> time
> > >> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here
it
> > >> > > > > > > > >>>>>>>>> is
> > using
> > >> a
> > >> > > > sample
> > >> > > > > > > > >>>>>>>>> data file you
> > >> > > > > > > > >>>>>>>>> sent to me. But remember I'm still
using
> > >> > > > > > > > >>>>>>>>> the
> > >> "old"
> > >> > > > wgrib2
> > >> > > > > > > > >>>>>>>>> so
> > >> > > > > > > > >>>>>>>>> I have that
> > >> > > > > > > > >>>>>>>>> funky forecast hour:
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > >> > > > > > > > >>>>>>>>>
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--
2147483624
> > >> > > > > > > > >>>>>>>>> hour
> > ave
> > >> > > > > > > > >>>>>>>>> fcst:vt=2018070505
> > >> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> > >> > fcst:vt=2018070605
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a
> > >> > > > > > > > >>>>>>>>> higher
> > >> > verbosity
> > >> > > > > > > > >>>>>>>>> level
> > >> > > > > > > > >>>>>>>>> (-v
> > >> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info
of
> > >> > > > > > > > >>>>>>>>> the
> > >> data
> > >> > it
> > >> > > is
> > >> > > > > > > > >>>>>>>>> reading:
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> # Read the first record
> > >> > > > > > > > >>>>>>>>> plot_data_plane
> > >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v
4
> > >> > > > > > > > >>>>>>>>> ...
> > >> > > > > > > > >>>>>>>>> DEBUG 4: valid time:
20180705_050000
> > >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > >> > > > > > > > >>>>>>>>> DEBUG 4: init time:
20180704_060000
> > >> > > > > > > > >>>>>>>>> ...
> > >> > > > > > > > >>>>>>>>> # Read the second record
> > >> > > > > > > > >>>>>>>>> plot_data_plane
> > >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v
4
> > >> > > > > > > > >>>>>>>>> ...
> > >> > > > > > > > >>>>>>>>> DEBUG 4: valid time:
20180706_050000
> > >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > >> > > > > > > > >>>>>>>>> DEBUG 4: init time:
20180704_060000
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> Hope that helps.
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> Thanks,
> > >> > > > > > > > >>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
> > >> > > > > > > > >>>>>>>>> Shafran -
> > >> NOAA
> > >> > > > > > > > >>>>>>>>> Affiliate <
> > >> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >> > > > > > > > >>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> Hi, John,
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> When I implemented all the changes
(and
> > >> > > > > > > > >>>>>>>>>> also
> > set
> > >> it
> > >> > up
> > >> > > > so
> > >> > > > > > > > >>>>>>>>>> it's
> > >> > > > > > > > >>>>>>>>>> reading the beta6 version), I used
this
> > command:
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > >> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1
> > >> > > > > > > > >>>>>>>>>> prepda.2018070912.nc
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > >
> > >> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > >> > > > > > > > >>>>>>>>>> -v 4
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> But then I got this error:
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> DEBUG 2:
> > >> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > >> > > > > > > > >>>>>>>>>> WARNING:
> > >> > > > > > > > >>>>>>>>>> WARNING:
> > >> > > > > > > > >>>>>>>>>> MetGrib2DataFile::data_plane_array() -
> > >> No
> > >> > > > > > > > >>>>>>>>>> matching
> > >> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > >> > > > > > > > >>>>>>>>>> WARNING:
> > >> > > > > > > > >>>>>>>>>> WARNING:
> > >> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() ->
no
> > fields
> > >> > > > matching
> > >> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > >> > > > > > > > >>>>>>>>>> WARNING:
> > >> > > > > > > > >>>>>>>>>> ERROR :
> > >> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() ->
no
> > >> requested
> > >> > > > > > > > >>>>>>>>>> forecast
> > >> > > > > > > > >>>>>>>>>> data
> > >> > > > > > > > >>>>>>>>>> found! Exiting...
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the
> > >> > > > > > > > >>>>>>>>>> wgrib2
> > >> output
> > >> > for
> > >> > > > the
> > >> > > > > > > > >>>>>>>>>> file
> > >> > > > > > > > >>>>>>>>>> in this case reads:
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> 14 (t14a2)
> > >> > > > > > > > >>>>>>>>>>
> > >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >> > > > >
> > >> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > >> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma
level:-7-
> > >> > > > > > > > >>>>>>>>>> 16
> > hour
> > >> ave
> > >> > > > fcst:
> > >> > > > > > > > >>>>>>>>>> 15 (t14a2)
> > >> > > > > > > > >>>>>>>>>>
> > >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be
> > >> > > > > > > > >>>>>>>>>> 20180708_04? You
> > >> had
> > >> > > > > > > > >>>>>>>>>> suggested
> > >> > > > > > > > >>>>>>>>>> that it should be the time stamp of
the
> > >> > > > > > > > >>>>>>>>>> 2nd
> > day,
> > >> but
> > >> > > > maybe
> > >> > > > > > > > >>>>>>>>>> it needs to be
> > >> > > > > > > > >>>>>>>>>> the time stamp of the first day?
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc
or
> > >> elsewhere.
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> Perry
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John
> > >> > > > > > > > >>>>>>>>>> Halley
> > >> Gotway <
> > >> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> Perry,
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air
> > >> > > > > > > > >>>>>>>>>>> quality vx
> > we
> > >> > did,
> > >> > > > we
> > >> > > > > > > > >>>>>>>>>>> used
> > >> > > > > > > > >>>>>>>>>>> observation files named:
> > >> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW
obs
> > >> > > > > > > > >>>>>>>>>>> and
> > >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM
obs
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> And those files don't exist in the
July
> > >> > > > > > > > >>>>>>>>>>> data
> > you
> > >> > just
> > >> > > > > > > > >>>>>>>>>>> sent.
> > >> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names
> > >> > > > > > > > >>>>>>>>>>> like:
> > >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> > >> > aqm.t12z.anowpm.pb.tm00
> > >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
> > >> aqm.t12z.anowpm.pb.tm120
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually
> > >> > > > > > > > >>>>>>>>>>> verifying
> > >> against
> > >> > the
> > >> > > > > > > > >>>>>>>>>>> BUFR
> > >> > > > > > > > >>>>>>>>>>> or
> > >> > > > > > > > >>>>>>>>>>> PREPBUFR.
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> Let me just tell you the key points
for
> > >> > > > > > > > >>>>>>>>>>> the
> > >> config
> > >> > > file
> > >> > > > > > > > >>>>>>>>>>> settings...
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need
to
> > >> > > > > > > > >>>>>>>>>>> pass 2
> > >> > input
> > >> > > > > > > > >>>>>>>>>>> files
> > >> > > > > > > > >>>>>>>>>>> -
> > >> > > > > > > > >>>>>>>>>>> thecurrent day and the previous
day...
> > >> > > > > > > > >>>>>>>>>>> pb2nc \
> > >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00
> > >> > > > > > > > >>>>>>>>>>> \
> > >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc
> > \
> > >> > > > > > > > >>>>>>>>>>> PB2NCConfig \
> > >> > > > > > > > >>>>>>>>>>> -pbfile
> > >> hourly.20180704/aqm.t12z.prepbufr.tm00
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up
the
> > >> > time_summary
> > >> > > > > > > > >>>>>>>>>>> like
> > >> > > > > > > > >>>>>>>>>>> this:
> > >> > > > > > > > >>>>>>>>>>> time_summary = {
> > >> > > > > > > > >>>>>>>>>>> flag = TRUE;
> > >> > > > > > > > >>>>>>>>>>> beg = "040000";
> > >> > > > > > > > >>>>>>>>>>> end = "040000";
> > >> > > > > > > > >>>>>>>>>>> step = 24*3600;
> > >> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600;
end
> > >> > > > > > > > >>>>>>>>>>> = 0; }
> > >> //
> > >> > > > > > > > >>>>>>>>>>> previous
> > >> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > >> > > > > > > > >>>>>>>>>>> grib_code = [];
> > >> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM"
];
> > >> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > >> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect
obs
> > >> > > > > > > > >>>>>>>>>>> every
> > >> hour
> > >> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
> > >> summaries
> > >> > > when
> > >> > > > at
> > >> > > > > > > > >>>>>>>>>>> least
> > >> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
> > >> > > > > > > > >>>>>>>>>>> }
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN
values
> > >> > > > > > > > >>>>>>>>>>> at
> > each
> > >> > > station
> > >> > > > > > > > >>>>>>>>>>> over the
> > >> > > > > > > > >>>>>>>>>>> defined time period. They're
identified
> > >> > > > > > > > >>>>>>>>>>> in
> > the
> > >> > > output
> > >> > > > by
> > >> > > > > > > > >>>>>>>>>>> the message type:
> > >> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or
AIRNOW_MEAN_230000
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup
the
> > "fcst"
> > >> and
> > >> > > > "obs"
> > >> > > > > > > > >>>>>>>>>>> sections like this for the max of the
1-
> > >> > > > > > > > >>>>>>>>>>> hour
> > >> data.
> > >> > > > > > > > >>>>>>>>>>> Instead
> > >> > > > > > > > >>>>>>>>>>> of specifying
> > >> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it
may
> > >> > > > > > > > >>>>>>>>>>> be more
> > >> > > > convenient
> > >> > > > > > > > >>>>>>>>>>> to specify the
> > >> > > > > > > > >>>>>>>>>>> valid time. So that you don't have
to
> > >> > > > > > > > >>>>>>>>>>> modify
> > >> the
> > >> > > > config
> > >> > > > > > > > >>>>>>>>>>> file, I'd suggest
> > >> > > > > > > > >>>>>>>>>>> using environment variables:
> > >> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> fcst = {
> > >> > > > > > > > >>>>>>>>>>> field = [
> > >> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > >> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > >> > > > > > > > >>>>>>>>>>> ];
> > >> > > > > > > > >>>>>>>>>>> }
> > >> > > > > > > > >>>>>>>>>>> obs = {
> > >> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > >> > > > > > > > >>>>>>>>>>> field = [
> > >> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
> > >> message_type =
> > >> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > >> > > > > > > > >>>>>>>>>>> ];
> > >> > > > > > > > >>>>>>>>>>> }
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
> > >> > > > > > > > >>>>>>>>>>> Halley
> > >> Gotway
> > >> > <
> > >> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > >> > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM
Perry
> > Shafran -
> > >> > NOAA
> > >> > > > > > > > >>>>>>>>>>>> Affiliate <
> > >> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >> > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore
all
> > >> > > > > > > > >>>>>>>>>>>>> the bc
> > >> grids
> > >> > > and
> > >> > > > > > > > >>>>>>>>>>>>> focus on
> > >> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some
time
> > >> > > > > > > > >>>>>>>>>>>>> to
> > copy
> > >> > files
> > >> > > > > > > > >>>>>>>>>>>>> over
> > >> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
> > >> > > > > > > > >>>>>>>>>>>>> let him work on the files he has
for
> > >> > > > > > > > >>>>>>>>>>>>> now and
> > >> if
> > >> > he
> > >> > > > runs
> > >> > > > > > > > >>>>>>>>>>>>> into problems, I'll
> > >> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
> > >> > > > > > > > >>>>>>>>>>>>> least
> > the
> > >> > 24-hr
> > >> > > > > > > > >>>>>>>>>>>>> max/avg files).
> > >> > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>> Thanks!
> > >> > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>> Perry
> > >> > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
> > jeff.mcqueen <
> > >> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus
raw
> > >> grids...not
> > >> > > for
> > >> > > > > > > > >>>>>>>>>>>>>> bias
> > >> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus
> > >> > > > > > > > >>>>>>>>>>>>>> grids, To
> > be
> > >> > > safe,
> > >> > > > > > > > >>>>>>>>>>>>>> you
> > >> > > > > > > > >>>>>>>>>>>>>> may want to provide
> > >> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
> > >> > > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran
-
> > >> > > > > > > > >>>>>>>>>>>>>> NOAA
> > >> > > Affiliate
> > >> > > > > > > > >>>>>>>>>>>>>> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent
over
> > >> > > > > > > > >>>>>>>>>>>>>> to
> > Theia
> > >> > for
> > >> > > > John
> > >> > > > > > > > >>>>>>>>>>>>>> to do
> > >> > > > > > > > >>>>>>>>>>>>>> some testing some production
files.
> > >> > > > > > > > >>>>>>>>>>>>>> Do you
> > >> > think
> > >> > > I
> > >> > > > > > > > >>>>>>>>>>>>>> should use parallels
> > >> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops
files
> > >> > > > > > > > >>>>>>>>>>>>>> are
> > OK
> > >> for
> > >> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > >> > > > > > > > >>>>>>>>>>>>>> for AQ?
> > >> > > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I
think
> > >> > > > > > > > >>>>>>>>>>>>>> that
> > the
> > >> ops
> > >> > > > files
> > >> > > > > > > > >>>>>>>>>>>>>> should
> > >> > > > > > > > >>>>>>>>>>>>>> be OK for now.
> > >> > > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>> Perry
> > >> > > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
> > jeff.mcqueen
> > >> <
> > >> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong
about
> > >> > > > > > > > >>>>>>>>>>>>>>> the
> > >> > > parallels.
> > >> > > > > > > > >>>>>>>>>>>>>>> We
> > >> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in
daily
> > >> > > > > > > > >>>>>>>>>>>>>>> max
> > hour
> > >> > > labels
> > >> > > > > > > > >>>>>>>>>>>>>>> but
> > >> > > > > > > > >>>>>>>>>>>>>>> management did not
> > >> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until
Q4FY19
> > >> > > > > > > > >>>>>>>>>>>>>>> CMAQ
> > >> > upgrade.
> > >> > > > > > > > >>>>>>>>>>>>>>> Jeff
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry
Shafran
> > >> > > > > > > > >>>>>>>>>>>>>>> - NOAA
> > >> > > > Affiliate
> > >> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels
have
> > >> > > > > > > > >>>>>>>>>>>>>>> the
> > >> > correct
> > >> > > > time
> > >> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better
if
> > >> > > > > > > > >>>>>>>>>>>>>>> I send
> > >> John
> > >> > > > over
> > >> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > >> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team
has
> > >> > > > > > > > >>>>>>>>>>>>>>> any
> > >> other
> > >> > > > > > > > >>>>>>>>>>>>>>> versions
> > >> > > > > > > > >>>>>>>>>>>>>>> of
> > >> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already
> > >> > > > > > > > >>>>>>>>>>>>>>> have -
> > but
> > >> > they
> > >> > > > can
> > >> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
> > >> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
> > >> > > > > > > > >>>>>>>>>>>>>>> wgrib2new
> > >> does
> > >> > > over
> > >> > > > > > > > >>>>>>>>>>>>>>> the
> > >> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> Thanks!
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> Perry
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> Perry
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
> > >> jeff.mcqueen <
> > >> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new,
you
> > >> > > > > > > > >>>>>>>>>>>>>>>> may
> > >> want
> > >> > to
> > >> > > > use
> > >> > > > > > > > >>>>>>>>>>>>>>>> the
> > >> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output
(para5).
> > >> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference,
I
> > >> > > > > > > > >>>>>>>>>>>>>>>> recall
> > >> > > Jianping
> > >> > > > > > > > >>>>>>>>>>>>>>>> also
> > >> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix
it
> > >> > > > > > > > >>>>>>>>>>>>>>>> but
> > >> > > management
> > >> > > > > > > > >>>>>>>>>>>>>>>> did
> > >> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > >> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have
to
> > >> > > > > > > > >>>>>>>>>>>>>>>> do so
> > >> with
> > >> > the
> > >> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
> > >> > > > > > > > >>>>>>>>>>>>>>>> implementation.
> > >> > > > > > > > >>>>>>>>>>>>>>>> Jeff
> > >> > > > > > > > >>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry
Shafran
> > >> > > > > > > > >>>>>>>>>>>>>>>> - NOAA
> > >> > > > Affiliate
> > >> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
> > >> conversation.
> > >> > I
> > >> > > > > > > > >>>>>>>>>>>>>>>> think
> > >> > > > > > > > >>>>>>>>>>>>>>>> I
> > >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the
> > >> > > > > > > > >>>>>>>>>>>>>>>> timestamps
> > some
> > >> > time
> > >> > > > ago,
> > >> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or
> > >> > > > > > > > >>>>>>>>>>>>>>>> with
> > >> Jeff. I
> > >> > > > think
> > >> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > >> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but
not
> > >> > > > > > > > >>>>>>>>>>>>>>>> too
> > >> > entirely
> > >> > > > > > > > >>>>>>>>>>>>>>>> sure.
> > >> > > > > > > > >>>>>>>>>>>>>>>> But he would have
> > >> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is
correct
> > >> > > > > > > > >>>>>>>>>>>>>>>> and
> > how
> > >> to
> > >> > > > > > > > >>>>>>>>>>>>>>>> approach
> > >> > > > > > > > >>>>>>>>>>>>>>>> this.
> > >> > > > > > > > >>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time
> > >> > > > > > > > >>>>>>>>>>>>>>>> stamp
> > >> issues
> > >> > > John
> > >> > > > > > > > >>>>>>>>>>>>>>>> talks
> > >> > > > > > > > >>>>>>>>>>>>>>>> about?
> > >> > > > > > > > >>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
> > >> > > > > > > > >>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>> Perry
> > >> > > > > > > > >>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM,
> > >> > > > > > > > >>>>>>>>>>>>>>>> John
> > Halley
> > >> > > > Gotway <
> > >> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all
> > >> > > > > > > > >>>>>>>>>>>>>>>>> squared
> > away
> > >> to
> > >> > > > > > > > >>>>>>>>>>>>>>>>> compute
> > >> > > > > > > > >>>>>>>>>>>>>>>>> the
> > >> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average
values
> > >> > > > > > > > >>>>>>>>>>>>>>>>> for
> > 1-
> > >> and
> > >> > > 8-
> > >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> > >> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > >> > \
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > >> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like
> > >> > > > > > > > >>>>>>>>>>>>>>>>> this:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > >> > > > > \
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>> PointStatConfig_AIRNOW
> > \
> > >> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config
files.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched
pairs:
> > >> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1
> > >> > > > > > > > >>>>>>>>>>>>>>>>> versus
> > >> COPO/A1,
> > >> > > for
> > >> > > > > > > > >>>>>>>>>>>>>>>>> observation type
> > >> > > > > > > > >>>>>>>>>>>>>>>>> AIRNOW_MAX_240000, over
> > >> > region
> > >> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
> > >> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > >> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147
> > >> > > > > > > > >>>>>>>>>>>>>>>>> pairs.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > >> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes
> > >> > > > > > > > >>>>>>>>>>>>>>>>> the
> > >> contents
> > >> > of
> > >> > > > > > > > >>>>>>>>>>>>>>>>> this
> > >> > > > > > > > >>>>>>>>>>>>>>>>> "max"
> > >> > > > > > > > >>>>>>>>>>>>>>>>> file:
> > >> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > >> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > >> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618
hour
> > >> > > > > > > > >>>>>>>>>>>>>>>>> ave
> > >> fcst:
> > >> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
> > >> > > > > > > > >>>>>>>>>>>>>>>>> sigma
> > >> > > > level:17-40
> > >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> > >> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty
> > >> > > > > > > > >>>>>>>>>>>>>>>>> funny. But
> > >> > > testing
> > >> > > > > > > > >>>>>>>>>>>>>>>>> revealed
> > >> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as
being
> > forecast
> > >> > > hours
> > >> > > > 17
> > >> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > >> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could
be
> > >> > > > > > > > >>>>>>>>>>>>>>>>> encoded
> > >> > > better.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast
hours
> > >> > > > > > > > >>>>>>>>>>>>>>>>> are in
> > >> the
> > >> > > same
> > >> > > > > > > > >>>>>>>>>>>>>>>>> file, we
> > >> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to
> > >> > > > > > > > >>>>>>>>>>>>>>>>> tell it
> > >> which
> > >> > > one
> > >> > > > > > > > >>>>>>>>>>>>>>>>> to
> > >> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > >> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > >> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > >> > > > > > > > >>>>>>>>>>>>>>>>> field = [
> > >> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1";
level
> > >> > > > > > > > >>>>>>>>>>>>>>>>> =
> > "L1";
> > >> > > > lead_time
> > >> > > > > > > > >>>>>>>>>>>>>>>>> =
> > >> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > >> > > > > > > > >>>>>>>>>>>>>>>>> ];
> > >> > > > > > > > >>>>>>>>>>>>>>>>> }
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and
observation
> > >> timestamps
> > >> > are
> > >> > > > off
> > >> > > > > > > > >>>>>>>>>>>>>>>>> by 1
> > >> > > > > > > > >>>>>>>>>>>>>>>>> hour.
> > >> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612
plus a
> > >> > > > > > > > >>>>>>>>>>>>>>>>> 41
> > hour
> > >> > lead
> > >> > > > > > > > >>>>>>>>>>>>>>>>> time
> > >> > > > > > > > >>>>>>>>>>>>>>>>> =
> > >> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > >> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute
a
> > >> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour
> > >> max
> > >> > > > value
> > >> > > > > > > > >>>>>>>>>>>>>>>>> from
> > >> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
> > >> > > > > > > > >>>>>>>>>>>>>>>>> valid at
> > >> 05Z
> > >> > > > while
> > >> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > >> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in
the
> > >> Point-Stat
> > >> > > > config
> > >> > > > > > > > >>>>>>>>>>>>>>>>> file by
> > >> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look
> > >> > > > > > > > >>>>>>>>>>>>>>>>> back 1
> > >> hour:
> > >> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > >> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > >> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > >> > > > > > > > >>>>>>>>>>>>>>>>> }
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > >> > > > > > > > >>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM
> > >> > > > > > > > >>>>>>>>>>>>>>>>> hsoh
> > >> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input
files
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> and
> > was
> > >> > > fixed.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> Halley
> > Gotway
> > >> > > wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the
time
> > >> summaries,
> > >> > > > but I
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> found
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> look on
> > >> > dakota
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> here:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of
pb2nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> to
> > >> compute
> > >> > > 24-
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> hour
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then
I
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> dumped
> > >> the
> > >> > > > output
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> commands
> > >> to
> > >> > > count
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> up
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> > >> > > pb2nc_output.txt
> > >> > > > |
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> wc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> > >> > > pb2nc_output.txt
> > >> > > > |
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> > >> > pb2nc_output.txt |
> > >> > > > wc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> > >> > pb2nc_output.txt |
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> sort
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> 114,538...
> > >> meaning
> > >> > > that
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> all
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> of
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> unique.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and
(4)
> > returns
> > >> > > 6,555
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> observations
> > >> are
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station
ID
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> in
> > that
> > >> > file
> > >> > > is
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> named
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
> > pb2nc_output.txt |
> > >> > grep
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> > >> 20180528_040000
> > >> > > 32.47
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> > >> 20180528_040000
> > >> > > 32.47
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> > >> 20180527_040000
> > >> > > 32.47
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> > >> 20180527_040000
> > >> > > 32.47
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> > >> 20180528_040000
> > >> > > 32.47
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> > >> 20180528_040000
> > >> > > 32.47
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines
are
> > >> repeated in
> > >> > > the
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> last two
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see
why
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> these
> > >> > > > duplicates
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> are
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39
PM
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> hsoh
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Halley
> > Gotway
> > >> > > wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense.
There's
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> no
> > >> reason
> > >> > why
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> have
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of
AOD
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> values
> > >> > > across 2
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files
from
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> which
> > we
> > >> > could
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33
PM
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > >> Soh
> > >> > <
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about
10
> > minutes
> > >> > > > (550.319
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> minutes.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I
will
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> not
> > >> check
> > >> > in
> > >> > > > the
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> files.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02
PM
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> hsoh
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the
output
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (the
> > >> > content
> > >> > > > of
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> for time
> > >> > > summary
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for
multiple
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> BURF
> > >> > inputs.
> > >> > > I
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> input
> > files
> > >> and
> > >> > > > will
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Halley
> > >> Gotway
> > >> > > > wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that
this
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> will
> > >> change
> > >> > > the
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this
case
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> not
> > >> > > currently
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I
add
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at
1:00
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> PM
> > Howard
> > >> > Soh
> > >> > > <
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> for time
> > >> > > summary
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in
to
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> SVN.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
> > >> /d3/personal/hsoh/data/pb2nc_summary/
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at
2:52
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> PM
> > John
> > >> > > Halley
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some
updates
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to the
> > >> > logic
> > >> > > > for
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test
trying
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > >> compute
> > >> > > > 24-hour
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO
from
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see
these
> > obs/hdr
> > >> > > > counts:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> (121832
> > >> > currently)
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through
the
> > Rscript
> > >> to
> > >> > > dump
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
> > AIRNOW_MAX_240000
> > >> > > string
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the
problem
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> is in
> > >> PB2NC
> > >> > > not
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> summaries... or
> > >> in
> > >> > the
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take
another
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> look
> > in
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
1:54
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PM
> > John
> > >> > > Halley
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging
into
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> this.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> 1:41 PM
> > >> Howard
> > >> > > > Soh <
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> observations
> > from
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> aqm.20180527/aqm.t12z.prepbufr.tm00
> > >> > > and
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > >> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem
for
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > obs
> > >> > > data.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 2:28 PM
> > >> John
> > >> > > > Halley
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu>
wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could
> > you
> > >> > > please
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example
of
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> running
> > >> > PB2NC.
> > >> > > > I
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over
to
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NCConfig \
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> request
> > >> > > 24-hourly
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to
04Z,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> like
> > >> this:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg =
"040000";
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end =
"040000";
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step =
24*3600;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
> > -24*3600;
> > >> > end =
> > >> > > > 0;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [
"COPO"
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> ];
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max"
];
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
> > observations...
> > >> > but
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which
is
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> not
> > >> what
> > >> > > you'd
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > >> > > > "AIRNOW_MAX_240000"
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as
you
> > pointed
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> referenced.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> problem
> > >> > caused
> > >> > > by
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files?
I
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> need 2
> > >> of
> > >> > > them
> > >> > > > to
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z
to
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 04Z
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take
a
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look?
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018
at
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 6:44 PM
> > >> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway
<johnhg at ucar.edu>
> > wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using
an
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW
> > >> > > PrepBUFR
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting
to
> > >> compute a
> > >> > 24
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> together next
> > >> > week.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018
at
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> 6:40
> > PM
> > >> > hsoh
> > >> > > <
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> script is
> > >> > > reasonable
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
> > observation
> > >> > data
> > >> > > in
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> index of
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the
NetCDF
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > >> was
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output
from
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc
> > >> > without
> > >> > > > any
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> has MORE
> > >> > > headers
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
> > 118193=headers).
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from the
> > >> > NetCDF
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ;
//
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (7295
> > >> > > currently)
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> > >> from 0
> > >> > > to
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1
(message
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> type
> > for
> > >> > > > summary)
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ
> > >> variable.
> > >> > > The
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index
for
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000"
is
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> bigger
> > >> than
> > >> > > > 3655.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script
seems
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> to be
> > OK.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
pntnc2ascii.R
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > >
> > >> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " |
sort
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MAX_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEAN_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ADPSFC_MEDIAN_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MIN_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_P80_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ADPSFC_RANGE_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_SDEV_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MAX_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEAN_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ADPUPA_MEDIAN_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MIN_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_P80_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ADPUPA_RANGE_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_SDEV_010000
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31
PM,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >> Halley
> > >> > > > Gotway
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take
a
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> look on
> > >> > dakota
> > >> > > > in
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script
to
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> dump
> > it
> > >> to
> > >> > > > ascii:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > >> > > > /d1/johnhg/MET/MET_development/
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > >
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> > >> > pb2nc_output.txt
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without
error
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> but
> > lists
> > >> > all
> > >> > > of
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW. But
> > >> some
> > >> > > of
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000...
as
> > >> evidenced
> > >> > by
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v
hdr_typ_table
> > >> > > > pb2nc_output.nc
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000"
;
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> indexing
> > or
> > >> > table
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the
script.
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> >
> >
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Tue Jul 09 16:25:43 2019
Perry,
This issue is fixed in met-8.1.1. Here's the GitHub issue documenting
it:
https://github.com/NCAR/MET/issues/1137
Thanks,
John
On Tue, Jul 9, 2019 at 3:38 PM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> I'm not sure that this has, indeed, been resolved. What's the
latest?
>
> Perry
>
> On Tue, Jul 9, 2019 at 5:24 PM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> > According to our records, your request has been resolved. If you
have any
> > further questions or concerns, please respond to this message.
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Julie Prestopnik
Time: Wed Jul 10 10:04:02 2019
Hi Perry.
I have updated the installation with met-8.1.1 on tide and luna.
Here is a link to our Existing Builds page for this release, so that
you
can see how to load, but essentially it is the same as before for met-
8.1
(see tide example below):
https://dtcenter.org/community-code/model-evaluation-tools-met/metv8-
1-existing-builds-metplus-2-1-installations
On tide:
module use /global/noscrub/Julie.Prestopnik/modulefiles
module load met/8.1
Please let us know how it goes.
Thanks,
Julie
On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
branch).
> It's not available yet for you. Julie will install it over the next
couple
> of days.
>
> Cheers,
> Howard
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Wed Jul 10 10:06:31 2019
Hi, Julie,
OK, great! I'll give this a try in the next day or so and let you all
know
what I see.
Perry
On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Perry.
>
> I have updated the installation with met-8.1.1 on tide and luna.
>
> Here is a link to our Existing Builds page for this release, so that
you
> can see how to load, but essentially it is the same as before for
met-8.1
> (see tide example below):
>
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-1-existing-builds-metplus-2-1-installations
>
> On tide:
>
> module use /global/noscrub/Julie.Prestopnik/modulefiles
> module load met/8.1
>
>
> Please let us know how it goes.
>
> Thanks,
> Julie
>
> On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
branch).
> > It's not available yet for you. Julie will install it over the
next
> couple
> > of days.
> >
> > Cheers,
> > Howard
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Wed Jul 10 10:07:22 2019
Are you going to make a general announcement that 8.1.1 is now
available
and what the fixes are? Or should I just refer people to the web
site?
Perry
On Wed, Jul 10, 2019 at 12:06 PM Perry Shafran - NOAA Affiliate <
perry.shafran at noaa.gov> wrote:
> Hi, Julie,
>
> OK, great! I'll give this a try in the next day or so and let you
all
> know what I see.
>
> Perry
>
> On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT <
> met_help at ucar.edu> wrote:
>
>> Hi Perry.
>>
>> I have updated the installation with met-8.1.1 on tide and luna.
>>
>> Here is a link to our Existing Builds page for this release, so
that you
>> can see how to load, but essentially it is the same as before for
met-8.1
>> (see tide example below):
>>
>> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-1-existing-builds-metplus-2-1-installations
>>
>> On tide:
>>
>> module use /global/noscrub/Julie.Prestopnik/modulefiles
>> module load met/8.1
>>
>>
>> Please let us know how it goes.
>>
>> Thanks,
>> Julie
>>
>> On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>> >
>> > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
branch).
>> > It's not available yet for you. Julie will install it over the
next
>> couple
>> > of days.
>> >
>> > Cheers,
>> > Howard
>> >
>>
>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Wed Jul 10 10:46:19 2019
Perry,
We sent out an email yesterday announcing the met-8.1.1 release. Did
you
receive it? I ask because with our recent web updates, we switched
over to
sending s/w announcements with a different system. All existing
registered
users "should" have been copied over. But if not, please let me know.
Here's the release notes from the website:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-8-1-1#notes
And details can be found in the GitHub issues for that release:
https://github.com/NCAR/MET/milestone/60?closed=1
Thanks,
John
On Wed, Jul 10, 2019 at 10:18 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Are you going to make a general announcement that 8.1.1 is now
available
> and what the fixes are? Or should I just refer people to the web
site?
>
> Perry
>
> On Wed, Jul 10, 2019 at 12:06 PM Perry Shafran - NOAA Affiliate <
> perry.shafran at noaa.gov> wrote:
>
> > Hi, Julie,
> >
> > OK, great! I'll give this a try in the next day or so and let you
all
> > know what I see.
> >
> > Perry
> >
> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Hi Perry.
> >>
> >> I have updated the installation with met-8.1.1 on tide and luna.
> >>
> >> Here is a link to our Existing Builds page for this release, so
that you
> >> can see how to load, but essentially it is the same as before for
> met-8.1
> >> (see tide example below):
> >>
> >>
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-1-existing-builds-metplus-2-1-installations
> >>
> >> On tide:
> >>
> >> module use /global/noscrub/Julie.Prestopnik/modulefiles
> >> module load met/8.1
> >>
> >>
> >> Please let us know how it goes.
> >>
> >> Thanks,
> >> Julie
> >>
> >> On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >> >
> >> > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
branch).
> >> > It's not available yet for you. Julie will install it over the
next
> >> couple
> >> > of days.
> >> >
> >> > Cheers,
> >> > Howard
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Wed Jul 10 10:47:32 2019
I must not be on the mailing list for whatever reason. I think I
usually
get the announcements, but then again, I don't remember.
Thanks!
Perry
On Wed, Jul 10, 2019 at 12:46 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Perry,
>
> We sent out an email yesterday announcing the met-8.1.1 release.
Did you
> receive it? I ask because with our recent web updates, we switched
over to
> sending s/w announcements with a different system. All existing
registered
> users "should" have been copied over. But if not, please let me
know.
>
> Here's the release notes from the website:
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-8-1-1#notes
>
> And details can be found in the GitHub issues for that release:
> https://github.com/NCAR/MET/milestone/60?closed=1
>
> Thanks,
> John
>
> On Wed, Jul 10, 2019 at 10:18 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > Are you going to make a general announcement that 8.1.1 is now
available
> > and what the fixes are? Or should I just refer people to the web
site?
> >
> > Perry
> >
> > On Wed, Jul 10, 2019 at 12:06 PM Perry Shafran - NOAA Affiliate <
> > perry.shafran at noaa.gov> wrote:
> >
> > > Hi, Julie,
> > >
> > > OK, great! I'll give this a try in the next day or so and let
you all
> > > know what I see.
> > >
> > > Perry
> > >
> > > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Hi Perry.
> > >>
> > >> I have updated the installation with met-8.1.1 on tide and
luna.
> > >>
> > >> Here is a link to our Existing Builds page for this release, so
that
> you
> > >> can see how to load, but essentially it is the same as before
for
> > met-8.1
> > >> (see tide example below):
> > >>
> > >>
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-1-existing-builds-metplus-2-1-installations
> > >>
> > >> On tide:
> > >>
> > >> module use /global/noscrub/Julie.Prestopnik/modulefiles
> > >> module load met/8.1
> > >>
> > >>
> > >> Please let us know how it goes.
> > >>
> > >> Thanks,
> > >> Julie
> > >>
> > >> On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > >> >
> > >> > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
> branch).
> > >> > It's not available yet for you. Julie will install it over
the next
> > >> couple
> > >> > of days.
> > >> >
> > >> > Cheers,
> > >> > Howard
> > >> >
> > >>
> > >>
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Wed Jul 10 10:54:26 2019
Here's the link to sign up for updates:
https://dtcenter.org/community-code/model-evaluation-tools-
met/subscribe
Thanks,
John
On Wed, Jul 10, 2019 at 10:47 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
> I must not be on the mailing list for whatever reason. I think I
usually
> get the announcements, but then again, I don't remember.
>
> Thanks!
>
> Perry
>
> On Wed, Jul 10, 2019 at 12:46 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Perry,
> >
> > We sent out an email yesterday announcing the met-8.1.1 release.
Did you
> > receive it? I ask because with our recent web updates, we
switched over
> to
> > sending s/w announcements with a different system. All existing
> registered
> > users "should" have been copied over. But if not, please let me
know.
> >
> > Here's the release notes from the website:
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-8-1-1#notes
> >
> > And details can be found in the GitHub issues for that release:
> > https://github.com/NCAR/MET/milestone/60?closed=1
> >
> > Thanks,
> > John
> >
> > On Wed, Jul 10, 2019 at 10:18 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >
> > > Are you going to make a general announcement that 8.1.1 is now
> available
> > > and what the fixes are? Or should I just refer people to the
web site?
> > >
> > > Perry
> > >
> > > On Wed, Jul 10, 2019 at 12:06 PM Perry Shafran - NOAA Affiliate
<
> > > perry.shafran at noaa.gov> wrote:
> > >
> > > > Hi, Julie,
> > > >
> > > > OK, great! I'll give this a try in the next day or so and let
you
> all
> > > > know what I see.
> > > >
> > > > Perry
> > > >
> > > > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Hi Perry.
> > > >>
> > > >> I have updated the installation with met-8.1.1 on tide and
luna.
> > > >>
> > > >> Here is a link to our Existing Builds page for this release,
so that
> > you
> > > >> can see how to load, but essentially it is the same as before
for
> > > met-8.1
> > > >> (see tide example below):
> > > >>
> > > >>
> > >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-1-existing-builds-metplus-2-1-installations
> > > >>
> > > >> On tide:
> > > >>
> > > >> module use /global/noscrub/Julie.Prestopnik/modulefiles
> > > >> module load met/8.1
> > > >>
> > > >>
> > > >> Please let us know how it goes.
> > > >>
> > > >> Thanks,
> > > >> Julie
> > > >>
> > > >> On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu
> >
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > > >> >
> > > >> > This issue was fixed in met-8.1.1
(bugfix_1137_pb2nc_hdr_vld
> > branch).
> > > >> > It's not available yet for you. Julie will install it over
the
> next
> > > >> couple
> > > >> > of days.
> > > >> >
> > > >> > Cheers,
> > > >> > Howard
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Thu Jul 25 11:27:38 2019
I think I am still getting this error, even in 8.1.1. Unless I am not
pointing to MET 8.1.1 correctly, but I do believe I set the module to
8.1.
Also the interesting thing is that it seems to work fine for the
OZMAX1 but
seems to be choking on the OZMAX8 (the 8-hr average). I'm wondering
if we
can take another look at this?
Thanks!
Perry
On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Perry.
>
> I have updated the installation with met-8.1.1 on tide and luna.
>
> Here is a link to our Existing Builds page for this release, so that
you
> can see how to load, but essentially it is the same as before for
met-8.1
> (see tide example below):
>
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-1-existing-builds-metplus-2-1-installations
>
> On tide:
>
> module use /global/noscrub/Julie.Prestopnik/modulefiles
> module load met/8.1
>
>
> Please let us know how it goes.
>
> Thanks,
> Julie
>
> On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
branch).
> > It's not available yet for you. Julie will install it over the
next
> couple
> > of days.
> >
> > Cheers,
> > Howard
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Howard Soh
Time: Thu Jul 25 13:34:31 2019
Would you upload the BURF file and PB2NC config files to the ftp site
or let us know the location we can get?
Cheers,
Howard
On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
> I think I am still getting this error, even in 8.1.1. Unless I am
not
> pointing to MET 8.1.1 correctly, but I do believe I set the module
to
> 8.1.
>
> Also the interesting thing is that it seems to work fine for the
> OZMAX1 but
> seems to be choking on the OZMAX8 (the 8-hr average). I'm wondering
> if we
> can take another look at this?
>
> Thanks!
>
> Perry
>
> On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
> <met_help at ucar.edu>
> wrote:
>
> > Hi Perry.
> >
> > I have updated the installation with met-8.1.1 on tide and luna.
> >
> > Here is a link to our Existing Builds page for this release, so
that
> > you
> > can see how to load, but essentially it is the same as before for
> > met-8.1
> > (see tide example below):
> >
> > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > 1-existing-builds-metplus-2-1-installations
> >
> > On tide:
> >
> > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > module load met/8.1
> >
> >
> > Please let us know how it goes.
> >
> > Thanks,
> > Julie
> >
> > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >
> > > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
> > > branch).
> > > It's not available yet for you. Julie will install it over the
next
> > couple
> > > of days.
> > >
> > > Cheers,
> > > Howard
> > >
> >
> >
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Thu Jul 25 14:32:35 2019
Hi, Howard,
Do you have access to WCOSS?
Thanks!
Perry
On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT <met_help at ucar.edu>
wrote:
> Would you upload the BURF file and PB2NC config files to the ftp
site or
> let us know the location we can get?
>
> Cheers,
> Howard
>
> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
> > I think I am still getting this error, even in 8.1.1. Unless I am
not
> > pointing to MET 8.1.1 correctly, but I do believe I set the module
to
> > 8.1.
> >
> > Also the interesting thing is that it seems to work fine for the
> > OZMAX1 but
> > seems to be choking on the OZMAX8 (the 8-hr average). I'm
wondering
> > if we
> > can take another look at this?
> >
> > Thanks!
> >
> > Perry
> >
> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Perry.
> > >
> > > I have updated the installation with met-8.1.1 on tide and luna.
> > >
> > > Here is a link to our Existing Builds page for this release, so
that
> > > you
> > > can see how to load, but essentially it is the same as before
for
> > > met-8.1
> > > (see tide example below):
> > >
> > > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > > 1-existing-builds-metplus-2-1-installations
> > >
> > > On tide:
> > >
> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > > module load met/8.1
> > >
> > >
> > > Please let us know how it goes.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
> > > > branch).
> > > > It's not available yet for you. Julie will install it over the
next
> > > couple
> > > > of days.
> > > >
> > > > Cheers,
> > > > Howard
> > > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Jul 26 07:30:15 2019
Hi, Howard,
I put the data that you need online. Here are the commands that I
have
used:
pb2nc prepda.2019072412 prepda.2019072412.nc PB2NCConfig_aqmmax
-pbfile
prepda.2019072312
point_stat oz_DAY1 prepda.2019072412.nc PointStatConfig_AIRNOW_max -v
4
And here is the error I am getting:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for OZMAX8/L1.
DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0 climatology mean
levels,
and 0 climatology standard deviation levels.
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Searching 5110 observations from 5110 messages.
ERROR :
ERROR : StringArray::operator[](int) const -> range check error!
ERROR :
Thanks! It could simply be a case that I have not properly linked to
the
fixed version already installed on WCOSS.
Perry
On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA Affiliate <
perry.shafran at noaa.gov> wrote:
> Hi, Howard,
>
> Do you have access to WCOSS?
>
> Thanks!
>
> Perry
>
> On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT
<met_help at ucar.edu>
> wrote:
>
>> Would you upload the BURF file and PB2NC config files to the ftp
site or
>> let us know the location we can get?
>>
>> Cheers,
>> Howard
>>
>> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
>> > I think I am still getting this error, even in 8.1.1. Unless I
am not
>> > pointing to MET 8.1.1 correctly, but I do believe I set the
module to
>> > 8.1.
>> >
>> > Also the interesting thing is that it seems to work fine for the
>> > OZMAX1 but
>> > seems to be choking on the OZMAX8 (the 8-hr average). I'm
wondering
>> > if we
>> > can take another look at this?
>> >
>> > Thanks!
>> >
>> > Perry
>> >
>> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
>> > <met_help at ucar.edu>
>> > wrote:
>> >
>> > > Hi Perry.
>> > >
>> > > I have updated the installation with met-8.1.1 on tide and
luna.
>> > >
>> > > Here is a link to our Existing Builds page for this release, so
that
>> > > you
>> > > can see how to load, but essentially it is the same as before
for
>> > > met-8.1
>> > > (see tide example below):
>> > >
>> > > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
>> > > 1-existing-builds-metplus-2-1-installations
>> > >
>> > > On tide:
>> > >
>> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
>> > > module load met/8.1
>> > >
>> > >
>> > > Please let us know how it goes.
>> > >
>> > > Thanks,
>> > > Julie
>> > >
>> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
>> > > >
>> > > > This issue was fixed in met-8.1.1 (bugfix_1137_pb2nc_hdr_vld
>> > > > branch).
>> > > > It's not available yet for you. Julie will install it over
the next
>> > > couple
>> > > > of days.
>> > > >
>> > > > Cheers,
>> > > > Howard
>> > > >
>> > >
>> > >
>>
>>
>>
>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Julie Prestopnik
Time: Fri Jul 26 08:41:21 2019
Hi Perry.
To check the version that you are running on WCOSS, you can type the
tool's
name followed by --version. For example:
"pb2nc --version"
That should return:
MET Version: V8.1.1
Repository: https://github.com/NCAR/MET
Last Changed Rev: e4fcc77
Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
if you are using met-8.1.1.
I hope that helps.
Julie
On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, Howard,
>
> I put the data that you need online. Here are the commands that I
have
> used:
>
> pb2nc prepda.2019072412 prepda.2019072412.nc PB2NCConfig_aqmmax
-pbfile
> prepda.2019072312
> point_stat oz_DAY1 prepda.2019072412.nc PointStatConfig_AIRNOW_max
-v 4
>
> And here is the error I am getting:
>
> DEBUG 2:
>
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for OZMAX8/L1.
> DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0 climatology mean
levels,
> and 0 climatology standard deviation levels.
> DEBUG 2:
> DEBUG 2:
>
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Searching 5110 observations from 5110 messages.
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> Thanks! It could simply be a case that I have not properly linked
to the
> fixed version already installed on WCOSS.
>
> Perry
>
> On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA Affiliate <
> perry.shafran at noaa.gov> wrote:
>
> > Hi, Howard,
> >
> > Do you have access to WCOSS?
> >
> > Thanks!
> >
> > Perry
> >
> > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> >> Would you upload the BURF file and PB2NC config files to the ftp
site or
> >> let us know the location we can get?
> >>
> >> Cheers,
> >> Howard
> >>
> >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
> >> > I think I am still getting this error, even in 8.1.1. Unless I
am not
> >> > pointing to MET 8.1.1 correctly, but I do believe I set the
module to
> >> > 8.1.
> >> >
> >> > Also the interesting thing is that it seems to work fine for
the
> >> > OZMAX1 but
> >> > seems to be choking on the OZMAX8 (the 8-hr average). I'm
wondering
> >> > if we
> >> > can take another look at this?
> >> >
> >> > Thanks!
> >> >
> >> > Perry
> >> >
> >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
> >> > <met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi Perry.
> >> > >
> >> > > I have updated the installation with met-8.1.1 on tide and
luna.
> >> > >
> >> > > Here is a link to our Existing Builds page for this release,
so that
> >> > > you
> >> > > can see how to load, but essentially it is the same as before
for
> >> > > met-8.1
> >> > > (see tide example below):
> >> > >
> >> > >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> >> > > 1-existing-builds-metplus-2-1-installations
> >> > >
> >> > > On tide:
> >> > >
> >> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> >> > > module load met/8.1
> >> > >
> >> > >
> >> > > Please let us know how it goes.
> >> > >
> >> > > Thanks,
> >> > > Julie
> >> > >
> >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<met_help at ucar.edu
> >
> >> > > wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >> > > >
> >> > > > This issue was fixed in met-8.1.1
(bugfix_1137_pb2nc_hdr_vld
> >> > > > branch).
> >> > > > It's not available yet for you. Julie will install it over
the
> next
> >> > > couple
> >> > > > of days.
> >> > > >
> >> > > > Cheers,
> >> > > > Howard
> >> > > >
> >> > >
> >> > >
> >>
> >>
> >>
> >>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Howard Soh
Time: Fri Jul 26 08:49:59 2019
I was not able to reproduce the problem.
Maybe two possible ways to check if 8.1.1 or not.
Option 1: the timestamp of pb2nc?
stat `which pb2nc`
OR
stat $(which pb2nc)
Option 2: Check if the other problem exists which was fixed with
8.1.1,
for example, time slicing by using 'yyyymmdd_hhmmss' instead of
offset.
Here is an example:
plot_data_plane test.nc test.ps 'name="tmp"; level="(0,*,*)";' -v 4
DEBUG 4: valid time: 20120101_210000
==>
plot_data_plane test.nc test.ps 'name="tmp";
level="(20120101_210000,*,*)";' -v 4
Cheers,
Howard
hsoh@@dakota:RT_89921_pb2nc_TimeSummary$ export OBS_BUFR_VAR='"COPO"'
hsoh@@dakota:RT_89921_pb2nc_TimeSummary$ /usr/local/met-
8.1/bin/pb2nc/pb2nc prepda.2019072412 prepda.2019072412.nc
PB2NCConfig_aqmmax -pbfile prepda.2019072312
DEBUG 1: Default Config File:
/d3/personal/hsoh/svn_latest/met/share/met/config/PB2NCConfig_default
DEBUG 1: User Config File: PB2NCConfig_aqmmax
DEBUG 1: Creating NetCDF File: prepda.2019072412.nc
DEBUG 1: Pre-processing BUFR File for variable names from
prepda.2019072312
DEBUG 1: process_pbfile_metadata() -> the number of records: 65747
DEBUG 1: Processing Bufr File: prepda.2019072312
DEBUG 1: Blocking Bufr file to: /tmp/tmp_pb2nc_blk_23634_0
DEBUG 2: Processing 65747 Bufr messages...
DEBUG 2: Bufr Time Center: 20190723_120000
DEBUG 2: Searching Time Window: 20190722_120000 to
20190724_120000
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90%
95% 100% 100%
DEBUG 2: Total Messages processed = 65747
DEBUG 2: Rejected based on message type = 0
DEBUG 2: Rejected based on station id = 0
DEBUG 2: Rejected based on valid time = 0
DEBUG 2: Rejected based on masking grid = 0
DEBUG 2: Rejected based on masking polygon = 0
DEBUG 2: Rejected based on elevation = 0
DEBUG 2: Rejected based on pb report type = 0
DEBUG 2: Rejected based on input report type = 0
DEBUG 2: Rejected based on instrument type = 0
DEBUG 2: Rejected based on zero observations = 4802
DEBUG 2: Total Messages retained = 60945
DEBUG 2: Total observations retained or derived = 60945
DEBUG 1: Obs time between 20190723_010000 and 20190724_000000
DEBUG 1: Pre-processing BUFR File for variable names from
prepda.2019072412
DEBUG 1: process_pbfile_metadata() -> the number of records: 65760
DEBUG 1: Processing Bufr File: prepda.2019072412
DEBUG 1: Blocking Bufr file to: /tmp/tmp_pb2nc_blk_23634_0
DEBUG 2: Processing 65760 Bufr messages...
DEBUG 2: Bufr Time Center: 20190724_120000
DEBUG 2: Searching Time Window: 20190723_120000 to
20190725_120000
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90%
95% 100% 100%
DEBUG 2: Total Messages processed = 65760
DEBUG 2: Rejected based on message type = 0
DEBUG 2: Rejected based on station id = 0
DEBUG 2: Rejected based on valid time = 0
DEBUG 2: Rejected based on masking grid = 0
DEBUG 2: Rejected based on masking polygon = 0
DEBUG 2: Rejected based on elevation = 0
DEBUG 2: Rejected based on pb report type = 0
DEBUG 2: Rejected based on input report type = 0
DEBUG 2: Rejected based on instrument type = 0
DEBUG 2: Rejected based on zero observations = 4696
DEBUG 2: Total Messages retained = 61064
DEBUG 2: Total observations retained or derived = 61064
DEBUG 1: Obs time between 20190724_010000 and 20190725_000000
hsoh@@dakota:/d3/personal/hsoh/data/met_help/RT_89921_pb2nc_TimeSummary$
ll -rt
total 4488
-rw-rw-r-- 1 hsoh rap 2119512 Jul 26 08:01 prepda.2019072312
-rw-rw-r-- 1 hsoh rap 2119928 Jul 26 08:04 prepda.2019072412
-rw-rw-r-- 1 hsoh rap 3873 Jul 26 08:04 PB2NCConfig_aqmmax
-rw-rw-r-- 1 hsoh rap 4981 Jul 26 08:04 PointStatConfig_AIRNOW_max
-rw-rw-r-- 1 hsoh rap 0 Jul 26 08:07 met_config_23500_0
-rw-rw-r-- 1 hsoh rap 1659 Jul 26 08:10 met_config_23587_0
-rw-rw-r-- 1 hsoh rap 1659 Jul 26 08:10 met_config_23590_0
-rw-rw-r-- 1 hsoh rap 318848 Jul 26 08:14 prepda.2019072412.nc
On Fri Jul 26 07:30:15 2019, perry.shafran at noaa.gov wrote:
> Hi, Howard,
>
> I put the data that you need online. Here are the commands that I
> have
> used:
>
> pb2nc prepda.2019072412 prepda.2019072412.nc PB2NCConfig_aqmmax
> -pbfile
> prepda.2019072312
> point_stat oz_DAY1 prepda.2019072412.nc PointStatConfig_AIRNOW_max
-v
> 4
>
> And here is the error I am getting:
>
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for OZMAX8/L1.
> DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0 climatology mean
> levels,
> and 0 climatology standard deviation levels.
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Searching 5110 observations from 5110 messages.
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> Thanks! It could simply be a case that I have not properly linked
to
> the
> fixed version already installed on WCOSS.
>
> Perry
>
> On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA Affiliate <
> perry.shafran at noaa.gov> wrote:
>
> > Hi, Howard,
> >
> > Do you have access to WCOSS?
> >
> > Thanks!
> >
> > Perry
> >
> > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT
<met_help at ucar.edu>
> > wrote:
> >
> >> Would you upload the BURF file and PB2NC config files to the ftp
> >> site or
> >> let us know the location we can get?
> >>
> >> Cheers,
> >> Howard
> >>
> >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
> >> > I think I am still getting this error, even in 8.1.1. Unless I
am
> >> > not
> >> > pointing to MET 8.1.1 correctly, but I do believe I set the
module
> >> > to
> >> > 8.1.
> >> >
> >> > Also the interesting thing is that it seems to work fine for
the
> >> > OZMAX1 but
> >> > seems to be choking on the OZMAX8 (the 8-hr average). I'm
> >> > wondering
> >> > if we
> >> > can take another look at this?
> >> >
> >> > Thanks!
> >> >
> >> > Perry
> >> >
> >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
> >> > <met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi Perry.
> >> > >
> >> > > I have updated the installation with met-8.1.1 on tide and
luna.
> >> > >
> >> > > Here is a link to our Existing Builds page for this release,
so
> >> > > that
> >> > > you
> >> > > can see how to load, but essentially it is the same as before
> >> > > for
> >> > > met-8.1
> >> > > (see tide example below):
> >> > >
> >> > > https://dtcenter.org/community-code/model-evaluation-tools-
> >> > > met/metv8-
> >> > > 1-existing-builds-metplus-2-1-installations
> >> > >
> >> > > On tide:
> >> > >
> >> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> >> > > module load met/8.1
> >> > >
> >> > >
> >> > > Please let us know how it goes.
> >> > >
> >> > > Thanks,
> >> > > Julie
> >> > >
> >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
> >> > > <met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> >> > > > >
> >> > > >
> >> > > > This issue was fixed in met-8.1.1
(bugfix_1137_pb2nc_hdr_vld
> >> > > > branch).
> >> > > > It's not available yet for you. Julie will install it over
the
> >> > > > next
> >> > > couple
> >> > > > of days.
> >> > > >
> >> > > > Cheers,
> >> > > > Howard
> >> > > >
> >> > >
> >> > >
> >>
> >>
> >>
> >>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Jul 26 09:47:15 2019
Hi, Julie,
Seems like it's 8.1 and not 8.1.1. What's the best way to access
8.1.1 if
the module is still 8.1?
Thanks!
Perry
On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Perry.
>
> To check the version that you are running on WCOSS, you can type the
tool's
> name followed by --version. For example:
>
> "pb2nc --version"
>
> That should return:
>
> MET Version: V8.1.1
> Repository: https://github.com/NCAR/MET
> Last Changed Rev: e4fcc77
> Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
>
> if you are using met-8.1.1.
>
> I hope that helps.
>
> Julie
>
> On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > Hi, Howard,
> >
> > I put the data that you need online. Here are the commands that I
have
> > used:
> >
> > pb2nc prepda.2019072412 prepda.2019072412.nc PB2NCConfig_aqmmax
-pbfile
> > prepda.2019072312
> > point_stat oz_DAY1 prepda.2019072412.nc
PointStatConfig_AIRNOW_max -v 4
> >
> > And here is the error I am getting:
> >
> > DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 2: Reading data for OZMAX8/L1.
> > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0 climatology mean
> levels,
> > and 0 climatology standard deviation levels.
> > DEBUG 2:
> > DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 2: Searching 5110 observations from 5110 messages.
> > ERROR :
> > ERROR : StringArray::operator[](int) const -> range check error!
> > ERROR :
> >
> > Thanks! It could simply be a case that I have not properly linked
to the
> > fixed version already installed on WCOSS.
> >
> > Perry
> >
> > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA Affiliate <
> > perry.shafran at noaa.gov> wrote:
> >
> > > Hi, Howard,
> > >
> > > Do you have access to WCOSS?
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > >> Would you upload the BURF file and PB2NC config files to the
ftp site
> or
> > >> let us know the location we can get?
> > >>
> > >> Cheers,
> > >> Howard
> > >>
> > >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
> > >> > I think I am still getting this error, even in 8.1.1. Unless
I am
> not
> > >> > pointing to MET 8.1.1 correctly, but I do believe I set the
module
> to
> > >> > 8.1.
> > >> >
> > >> > Also the interesting thing is that it seems to work fine for
the
> > >> > OZMAX1 but
> > >> > seems to be choking on the OZMAX8 (the 8-hr average). I'm
wondering
> > >> > if we
> > >> > can take another look at this?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > Perry
> > >> >
> > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
> > >> > <met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Hi Perry.
> > >> > >
> > >> > > I have updated the installation with met-8.1.1 on tide and
luna.
> > >> > >
> > >> > > Here is a link to our Existing Builds page for this
release, so
> that
> > >> > > you
> > >> > > can see how to load, but essentially it is the same as
before for
> > >> > > met-8.1
> > >> > > (see tide example below):
> > >> > >
> > >> > >
> > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > >> > > 1-existing-builds-metplus-2-1-installations
> > >> > >
> > >> > > On tide:
> > >> > >
> > >> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > >> > > module load met/8.1
> > >> > >
> > >> > >
> > >> > > Please let us know how it goes.
> > >> > >
> > >> > > Thanks,
> > >> > > Julie
> > >> > >
> > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <
> met_help at ucar.edu
> > >
> > >> > > wrote:
> > >> > >
> > >> > > >
> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >> > > >
> > >> > > > This issue was fixed in met-8.1.1
(bugfix_1137_pb2nc_hdr_vld
> > >> > > > branch).
> > >> > > > It's not available yet for you. Julie will install it
over the
> > next
> > >> > > couple
> > >> > > > of days.
> > >> > > >
> > >> > > > Cheers,
> > >> > > > Howard
> > >> > > >
> > >> > >
> > >> > >
> > >>
> > >>
> > >>
> > >>
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Julie Prestopnik
Time: Fri Jul 26 10:00:09 2019
Hi Perry.
I'm not sure what module you are loading or what machine you're using,
but
when I run the following, I can see that I am using met-8.1.1. on
tide:
[Julie.Prestopnik at t21a2 ~]$ module use
/global/noscrub/Julie.Prestopnik/modulefiles
[Julie.Prestopnik at t21a2 ~]$ module load met/8.1
[Julie.Prestopnik at t21a2 ~]$ pb2nc --version
MET Version: V8.1.1
Repository: https://github.com/NCAR/MET
Last Changed Rev: e4fcc77
Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
Please let me know if you have any questions.
Julie
On Fri, Jul 26, 2019 at 9:47 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, Julie,
>
> Seems like it's 8.1 and not 8.1.1. What's the best way to access
8.1.1 if
> the module is still 8.1?
>
> Thanks!
>
> Perry
>
> On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi Perry.
> >
> > To check the version that you are running on WCOSS, you can type
the
> tool's
> > name followed by --version. For example:
> >
> > "pb2nc --version"
> >
> > That should return:
> >
> > MET Version: V8.1.1
> > Repository: https://github.com/NCAR/MET
> > Last Changed Rev: e4fcc77
> > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> >
> > if you are using met-8.1.1.
> >
> > I hope that helps.
> >
> > Julie
> >
> > On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >
> > > Hi, Howard,
> > >
> > > I put the data that you need online. Here are the commands that
I have
> > > used:
> > >
> > > pb2nc prepda.2019072412 prepda.2019072412.nc PB2NCConfig_aqmmax
> -pbfile
> > > prepda.2019072312
> > > point_stat oz_DAY1 prepda.2019072412.nc
PointStatConfig_AIRNOW_max
> -v 4
> > >
> > > And here is the error I am getting:
> > >
> > > DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > DEBUG 2:
> > > DEBUG 2: Reading data for OZMAX8/L1.
> > > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0 climatology
mean
> > levels,
> > > and 0 climatology standard deviation levels.
> > > DEBUG 2:
> > > DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > DEBUG 2:
> > > DEBUG 2: Searching 5110 observations from 5110 messages.
> > > ERROR :
> > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > ERROR :
> > >
> > > Thanks! It could simply be a case that I have not properly
linked to
> the
> > > fixed version already installed on WCOSS.
> > >
> > > Perry
> > >
> > > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA Affiliate <
> > > perry.shafran at noaa.gov> wrote:
> > >
> > > > Hi, Howard,
> > > >
> > > > Do you have access to WCOSS?
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT
<met_help at ucar.edu
> >
> > > > wrote:
> > > >
> > > >> Would you upload the BURF file and PB2NC config files to the
ftp
> site
> > or
> > > >> let us know the location we can get?
> > > >>
> > > >> Cheers,
> > > >> Howard
> > > >>
> > > >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
> > > >> > I think I am still getting this error, even in 8.1.1.
Unless I am
> > not
> > > >> > pointing to MET 8.1.1 correctly, but I do believe I set the
module
> > to
> > > >> > 8.1.
> > > >> >
> > > >> > Also the interesting thing is that it seems to work fine
for the
> > > >> > OZMAX1 but
> > > >> > seems to be choking on the OZMAX8 (the 8-hr average). I'm
> wondering
> > > >> > if we
> > > >> > can take another look at this?
> > > >> >
> > > >> > Thanks!
> > > >> >
> > > >> > Perry
> > > >> >
> > > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
> > > >> > <met_help at ucar.edu>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Perry.
> > > >> > >
> > > >> > > I have updated the installation with met-8.1.1 on tide
and luna.
> > > >> > >
> > > >> > > Here is a link to our Existing Builds page for this
release, so
> > that
> > > >> > > you
> > > >> > > can see how to load, but essentially it is the same as
before
> for
> > > >> > > met-8.1
> > > >> > > (see tide example below):
> > > >> > >
> > > >> > >
> > > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > > >> > > 1-existing-builds-metplus-2-1-installations
> > > >> > >
> > > >> > > On tide:
> > > >> > >
> > > >> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > > >> > > module load met/8.1
> > > >> > >
> > > >> > >
> > > >> > > Please let us know how it goes.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Julie
> > > >> > >
> > > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <
> > met_help at ucar.edu
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> >
> > > >> > > >
> > > >> > > > This issue was fixed in met-8.1.1
(bugfix_1137_pb2nc_hdr_vld
> > > >> > > > branch).
> > > >> > > > It's not available yet for you. Julie will install it
over the
> > > next
> > > >> > > couple
> > > >> > > > of days.
> > > >> > > >
> > > >> > > > Cheers,
> > > >> > > > Howard
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Jul 26 10:16:14 2019
I have this in my .bashrc:
module use /global/noscrub/Julie.Prestopnik/modulefiles
module load met/8.1
Yet I seem to be stuck on 8.1.
Plus I am getting this error:
23 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts > module list
Currently Loaded Modulefiles:
ModuleCmd_List.c(146):FATAL:996: The environment variables
LOADEDMODULES
and _LMFILES_ have inconsistent lengths.
24 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts
So I'm not sure what I'm doing wrong here.
Perry
On Fri, Jul 26, 2019 at 12:00 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Perry.
>
> I'm not sure what module you are loading or what machine you're
using, but
> when I run the following, I can see that I am using met-8.1.1. on
tide:
>
> [Julie.Prestopnik at t21a2 ~]$ module use
> /global/noscrub/Julie.Prestopnik/modulefiles
>
> [Julie.Prestopnik at t21a2 ~]$ module load met/8.1
>
> [Julie.Prestopnik at t21a2 ~]$ pb2nc --version
>
>
> MET Version: V8.1.1
>
> Repository: https://github.com/NCAR/MET
>
> Last Changed Rev: e4fcc77
>
> Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
>
>
> Please let me know if you have any questions.
>
> Julie
>
> On Fri, Jul 26, 2019 at 9:47 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > Hi, Julie,
> >
> > Seems like it's 8.1 and not 8.1.1. What's the best way to access
8.1.1
> if
> > the module is still 8.1?
> >
> > Thanks!
> >
> > Perry
> >
> > On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Hi Perry.
> > >
> > > To check the version that you are running on WCOSS, you can type
the
> > tool's
> > > name followed by --version. For example:
> > >
> > > "pb2nc --version"
> > >
> > > That should return:
> > >
> > > MET Version: V8.1.1
> > > Repository: https://github.com/NCAR/MET
> > > Last Changed Rev: e4fcc77
> > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > >
> > > if you are using met-8.1.1.
> > >
> > > I hope that helps.
> > >
> > > Julie
> > >
> > > On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > Hi, Howard,
> > > >
> > > > I put the data that you need online. Here are the commands
that I
> have
> > > > used:
> > > >
> > > > pb2nc prepda.2019072412 prepda.2019072412.nc
PB2NCConfig_aqmmax
> > -pbfile
> > > > prepda.2019072312
> > > > point_stat oz_DAY1 prepda.2019072412.nc
PointStatConfig_AIRNOW_max
> > -v 4
> > > >
> > > > And here is the error I am getting:
> > > >
> > > > DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > DEBUG 2:
> > > > DEBUG 2: Reading data for OZMAX8/L1.
> > > > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0 climatology
mean
> > > levels,
> > > > and 0 climatology standard deviation levels.
> > > > DEBUG 2:
> > > > DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > DEBUG 2:
> > > > DEBUG 2: Searching 5110 observations from 5110 messages.
> > > > ERROR :
> > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > ERROR :
> > > >
> > > > Thanks! It could simply be a case that I have not properly
linked to
> > the
> > > > fixed version already installed on WCOSS.
> > > >
> > > > Perry
> > > >
> > > > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA Affiliate
<
> > > > perry.shafran at noaa.gov> wrote:
> > > >
> > > > > Hi, Howard,
> > > > >
> > > > > Do you have access to WCOSS?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Perry
> > > > >
> > > > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT <
> met_help at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > >> Would you upload the BURF file and PB2NC config files to
the ftp
> > site
> > > or
> > > > >> let us know the location we can get?
> > > > >>
> > > > >> Cheers,
> > > > >> Howard
> > > > >>
> > > > >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov wrote:
> > > > >> > I think I am still getting this error, even in 8.1.1.
Unless I
> am
> > > not
> > > > >> > pointing to MET 8.1.1 correctly, but I do believe I set
the
> module
> > > to
> > > > >> > 8.1.
> > > > >> >
> > > > >> > Also the interesting thing is that it seems to work fine
for the
> > > > >> > OZMAX1 but
> > > > >> > seems to be choking on the OZMAX8 (the 8-hr average).
I'm
> > wondering
> > > > >> > if we
> > > > >> > can take another look at this?
> > > > >> >
> > > > >> > Thanks!
> > > > >> >
> > > > >> > Perry
> > > > >> >
> > > > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via RT
> > > > >> > <met_help at ucar.edu>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Perry.
> > > > >> > >
> > > > >> > > I have updated the installation with met-8.1.1 on tide
and
> luna.
> > > > >> > >
> > > > >> > > Here is a link to our Existing Builds page for this
release,
> so
> > > that
> > > > >> > > you
> > > > >> > > can see how to load, but essentially it is the same as
before
> > for
> > > > >> > > met-8.1
> > > > >> > > (see tide example below):
> > > > >> > >
> > > > >> > >
> > > >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > > > >> > > 1-existing-builds-metplus-2-1-installations
> > > > >> > >
> > > > >> > > On tide:
> > > > >> > >
> > > > >> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > > > >> > > module load met/8.1
> > > > >> > >
> > > > >> > >
> > > > >> > > Please let us know how it goes.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Julie
> > > > >> > >
> > > > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <
> > > met_help at ucar.edu
> > > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > >
> > > > >> > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> > >
> > > > >> > > >
> > > > >> > > > This issue was fixed in met-8.1.1
(bugfix_1137_pb2nc_hdr_vld
> > > > >> > > > branch).
> > > > >> > > > It's not available yet for you. Julie will install it
over
> the
> > > > next
> > > > >> > > couple
> > > > >> > > > of days.
> > > > >> > > >
> > > > >> > > > Cheers,
> > > > >> > > > Howard
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Julie Prestopnik
Time: Fri Jul 26 10:26:43 2019
Hi Perry. Can you please open a new shell and let me know if that
error
still occurs if you type "module list"? Thanks!
Julie
On Fri, Jul 26, 2019 at 10:16 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> I have this in my .bashrc:
>
> module use /global/noscrub/Julie.Prestopnik/modulefiles
> module load met/8.1
>
> Yet I seem to be stuck on 8.1.
>
> Plus I am getting this error:
>
> 23 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts > module
list
> Currently Loaded Modulefiles:
> ModuleCmd_List.c(146):FATAL:996: The environment variables
LOADEDMODULES
> and _LMFILES_ have inconsistent lengths.
> 24 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts
>
> So I'm not sure what I'm doing wrong here.
>
> Perry
>
> On Fri, Jul 26, 2019 at 12:00 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi Perry.
> >
> > I'm not sure what module you are loading or what machine you're
using,
> but
> > when I run the following, I can see that I am using met-8.1.1. on
tide:
> >
> > [Julie.Prestopnik at t21a2 ~]$ module use
> > /global/noscrub/Julie.Prestopnik/modulefiles
> >
> > [Julie.Prestopnik at t21a2 ~]$ module load met/8.1
> >
> > [Julie.Prestopnik at t21a2 ~]$ pb2nc --version
> >
> >
> > MET Version: V8.1.1
> >
> > Repository: https://github.com/NCAR/MET
> >
> > Last Changed Rev: e4fcc77
> >
> > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> >
> >
> > Please let me know if you have any questions.
> >
> > Julie
> >
> > On Fri, Jul 26, 2019 at 9:47 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >
> > > Hi, Julie,
> > >
> > > Seems like it's 8.1 and not 8.1.1. What's the best way to
access 8.1.1
> > if
> > > the module is still 8.1?
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > > On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Perry.
> > > >
> > > > To check the version that you are running on WCOSS, you can
type the
> > > tool's
> > > > name followed by --version. For example:
> > > >
> > > > "pb2nc --version"
> > > >
> > > > That should return:
> > > >
> > > > MET Version: V8.1.1
> > > > Repository: https://github.com/NCAR/MET
> > > > Last Changed Rev: e4fcc77
> > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > > >
> > > > if you are using met-8.1.1.
> > > >
> > > > I hope that helps.
> > > >
> > > > Julie
> > > >
> > > > On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov via RT
<
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > > > >
> > > > > Hi, Howard,
> > > > >
> > > > > I put the data that you need online. Here are the commands
that I
> > have
> > > > > used:
> > > > >
> > > > > pb2nc prepda.2019072412 prepda.2019072412.nc
PB2NCConfig_aqmmax
> > > -pbfile
> > > > > prepda.2019072312
> > > > > point_stat oz_DAY1 prepda.2019072412.nc
> PointStatConfig_AIRNOW_max
> > > -v 4
> > > > >
> > > > > And here is the error I am getting:
> > > > >
> > > > > DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > DEBUG 2:
> > > > > DEBUG 2: Reading data for OZMAX8/L1.
> > > > > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0
climatology mean
> > > > levels,
> > > > > and 0 climatology standard deviation levels.
> > > > > DEBUG 2:
> > > > > DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > DEBUG 2:
> > > > > DEBUG 2: Searching 5110 observations from 5110 messages.
> > > > > ERROR :
> > > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > > ERROR :
> > > > >
> > > > > Thanks! It could simply be a case that I have not properly
linked
> to
> > > the
> > > > > fixed version already installed on WCOSS.
> > > > >
> > > > > Perry
> > > > >
> > > > > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA
Affiliate <
> > > > > perry.shafran at noaa.gov> wrote:
> > > > >
> > > > > > Hi, Howard,
> > > > > >
> > > > > > Do you have access to WCOSS?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Perry
> > > > > >
> > > > > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT <
> > met_help at ucar.edu
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Would you upload the BURF file and PB2NC config files to
the ftp
> > > site
> > > > or
> > > > > >> let us know the location we can get?
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Howard
> > > > > >>
> > > > > >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov
wrote:
> > > > > >> > I think I am still getting this error, even in 8.1.1.
Unless
> I
> > am
> > > > not
> > > > > >> > pointing to MET 8.1.1 correctly, but I do believe I set
the
> > module
> > > > to
> > > > > >> > 8.1.
> > > > > >> >
> > > > > >> > Also the interesting thing is that it seems to work
fine for
> the
> > > > > >> > OZMAX1 but
> > > > > >> > seems to be choking on the OZMAX8 (the 8-hr average).
I'm
> > > wondering
> > > > > >> > if we
> > > > > >> > can take another look at this?
> > > > > >> >
> > > > > >> > Thanks!
> > > > > >> >
> > > > > >> > Perry
> > > > > >> >
> > > > > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via
RT
> > > > > >> > <met_help at ucar.edu>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Perry.
> > > > > >> > >
> > > > > >> > > I have updated the installation with met-8.1.1 on
tide and
> > luna.
> > > > > >> > >
> > > > > >> > > Here is a link to our Existing Builds page for this
release,
> > so
> > > > that
> > > > > >> > > you
> > > > > >> > > can see how to load, but essentially it is the same
as
> before
> > > for
> > > > > >> > > met-8.1
> > > > > >> > > (see tide example below):
> > > > > >> > >
> > > > > >> > >
> > > > >
> > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > > > > >> > > 1-existing-builds-metplus-2-1-installations
> > > > > >> > >
> > > > > >> > > On tide:
> > > > > >> > >
> > > > > >> > > module use
/global/noscrub/Julie.Prestopnik/modulefiles
> > > > > >> > > module load met/8.1
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Please let us know how it goes.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Julie
> > > > > >> > >
> > > > > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> > > >
> > > > > >> > > >
> > > > > >> > > > This issue was fixed in met-8.1.1
> (bugfix_1137_pb2nc_hdr_vld
> > > > > >> > > > branch).
> > > > > >> > > > It's not available yet for you. Julie will install
it over
> > the
> > > > > next
> > > > > >> > > couple
> > > > > >> > > > of days.
> > > > > >> > > >
> > > > > >> > > > Cheers,
> > > > > >> > > > Howard
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Jul 26 10:28:01 2019
Yes, it does.
29 (t14a2) ~ > sh
1 (t14a2) ~ > module list
Currently Loaded Modulefiles:
ModuleCmd_List.c(146):FATAL:996: The environment variables
LOADEDMODULES
and _LMFILES_ have inconsistent lengths.
2 (t14a2) ~ >
Perry
On Fri, Jul 26, 2019 at 12:26 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Perry. Can you please open a new shell and let me know if that
error
> still occurs if you type "module list"? Thanks!
>
> Julie
>
> On Fri, Jul 26, 2019 at 10:16 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > I have this in my .bashrc:
> >
> > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > module load met/8.1
> >
> > Yet I seem to be stuck on 8.1.
> >
> > Plus I am getting this error:
> >
> > 23 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts > module
list
> > Currently Loaded Modulefiles:
> > ModuleCmd_List.c(146):FATAL:996: The environment variables
LOADEDMODULES
> > and _LMFILES_ have inconsistent lengths.
> > 24 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts
> >
> > So I'm not sure what I'm doing wrong here.
> >
> > Perry
> >
> > On Fri, Jul 26, 2019 at 12:00 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Hi Perry.
> > >
> > > I'm not sure what module you are loading or what machine you're
using,
> > but
> > > when I run the following, I can see that I am using met-8.1.1.
on tide:
> > >
> > > [Julie.Prestopnik at t21a2 ~]$ module use
> > > /global/noscrub/Julie.Prestopnik/modulefiles
> > >
> > > [Julie.Prestopnik at t21a2 ~]$ module load met/8.1
> > >
> > > [Julie.Prestopnik at t21a2 ~]$ pb2nc --version
> > >
> > >
> > > MET Version: V8.1.1
> > >
> > > Repository: https://github.com/NCAR/MET
> > >
> > > Last Changed Rev: e4fcc77
> > >
> > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > >
> > >
> > > Please let me know if you have any questions.
> > >
> > > Julie
> > >
> > > On Fri, Jul 26, 2019 at 9:47 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > Hi, Julie,
> > > >
> > > > Seems like it's 8.1 and not 8.1.1. What's the best way to
access
> 8.1.1
> > > if
> > > > the module is still 8.1?
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > > On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Perry.
> > > > >
> > > > > To check the version that you are running on WCOSS, you can
type
> the
> > > > tool's
> > > > > name followed by --version. For example:
> > > > >
> > > > > "pb2nc --version"
> > > > >
> > > > > That should return:
> > > > >
> > > > > MET Version: V8.1.1
> > > > > Repository: https://github.com/NCAR/MET
> > > > > Last Changed Rev: e4fcc77
> > > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > > > >
> > > > > if you are using met-8.1.1.
> > > > >
> > > > > I hope that helps.
> > > > >
> > > > > Julie
> > > > >
> > > > > On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov via
RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > > > > >
> > > > > > Hi, Howard,
> > > > > >
> > > > > > I put the data that you need online. Here are the
commands that
> I
> > > have
> > > > > > used:
> > > > > >
> > > > > > pb2nc prepda.2019072412 prepda.2019072412.nc
PB2NCConfig_aqmmax
> > > > -pbfile
> > > > > > prepda.2019072312
> > > > > > point_stat oz_DAY1 prepda.2019072412.nc
> > PointStatConfig_AIRNOW_max
> > > > -v 4
> > > > > >
> > > > > > And here is the error I am getting:
> > > > > >
> > > > > > DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > DEBUG 2:
> > > > > > DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0
climatology
> mean
> > > > > levels,
> > > > > > and 0 climatology standard deviation levels.
> > > > > > DEBUG 2:
> > > > > > DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > DEBUG 2:
> > > > > > DEBUG 2: Searching 5110 observations from 5110 messages.
> > > > > > ERROR :
> > > > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > > > ERROR :
> > > > > >
> > > > > > Thanks! It could simply be a case that I have not
properly
> linked
> > to
> > > > the
> > > > > > fixed version already installed on WCOSS.
> > > > > >
> > > > > > Perry
> > > > > >
> > > > > > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA
Affiliate <
> > > > > > perry.shafran at noaa.gov> wrote:
> > > > > >
> > > > > > > Hi, Howard,
> > > > > > >
> > > > > > > Do you have access to WCOSS?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Perry
> > > > > > >
> > > > > > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Would you upload the BURF file and PB2NC config files
to the
> ftp
> > > > site
> > > > > or
> > > > > > >> let us know the location we can get?
> > > > > > >>
> > > > > > >> Cheers,
> > > > > > >> Howard
> > > > > > >>
> > > > > > >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov
wrote:
> > > > > > >> > I think I am still getting this error, even in 8.1.1.
> Unless
> > I
> > > am
> > > > > not
> > > > > > >> > pointing to MET 8.1.1 correctly, but I do believe I
set the
> > > module
> > > > > to
> > > > > > >> > 8.1.
> > > > > > >> >
> > > > > > >> > Also the interesting thing is that it seems to work
fine for
> > the
> > > > > > >> > OZMAX1 but
> > > > > > >> > seems to be choking on the OZMAX8 (the 8-hr average).
I'm
> > > > wondering
> > > > > > >> > if we
> > > > > > >> > can take another look at this?
> > > > > > >> >
> > > > > > >> > Thanks!
> > > > > > >> >
> > > > > > >> > Perry
> > > > > > >> >
> > > > > > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik via
RT
> > > > > > >> > <met_help at ucar.edu>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Perry.
> > > > > > >> > >
> > > > > > >> > > I have updated the installation with met-8.1.1 on
tide and
> > > luna.
> > > > > > >> > >
> > > > > > >> > > Here is a link to our Existing Builds page for this
> release,
> > > so
> > > > > that
> > > > > > >> > > you
> > > > > > >> > > can see how to load, but essentially it is the same
as
> > before
> > > > for
> > > > > > >> > > met-8.1
> > > > > > >> > > (see tide example below):
> > > > > > >> > >
> > > > > > >> > >
> > > > > >
> > > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > > > > > >> > > 1-existing-builds-metplus-2-1-installations
> > > > > > >> > >
> > > > > > >> > > On tide:
> > > > > > >> > >
> > > > > > >> > > module use
/global/noscrub/Julie.Prestopnik/modulefiles
> > > > > > >> > > module load met/8.1
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Please let us know how it goes.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Julie
> > > > > > >> > >
> > > > > > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> > > > >
> > > > > > >> > > >
> > > > > > >> > > > This issue was fixed in met-8.1.1
> > (bugfix_1137_pb2nc_hdr_vld
> > > > > > >> > > > branch).
> > > > > > >> > > > It's not available yet for you. Julie will
install it
> over
> > > the
> > > > > > next
> > > > > > >> > > couple
> > > > > > >> > > > of days.
> > > > > > >> > > >
> > > > > > >> > > > Cheers,
> > > > > > >> > > > Howard
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: Julie Prestopnik
Time: Fri Jul 26 10:36:33 2019
Hi Perry.
I am no modulefile expert, but I believe there is perhaps a conflict
the in
the loaded modules. It could be a direct conflict with something
specified
in your .bashrc file or it could be something you're loading in your
.bashrc file and then something that MET or another module is loading.
For
example, the MET modulefile located
at /global/noscrub/Julie.Prestopnik/modulefiles/met/8.1 loads:
module load ics/15.0.3
module use /usrx/local/dev/modulefiles
module load python/2.7.14
Perhaps there is a problem with one of those? Usually, I get a
message
that says that one module is being swapped with another module, so I'm
not
sure why you're getting that error. If you're not able to determine
the
problem, I suggest talking with one of your IT folks who would likely
know
more about that than I. I hope that helps!
Julie
On Fri, Jul 26, 2019 at 10:28 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Yes, it does.
>
> 29 (t14a2) ~ > sh
> 1 (t14a2) ~ > module list
> Currently Loaded Modulefiles:
> ModuleCmd_List.c(146):FATAL:996: The environment variables
LOADEDMODULES
> and _LMFILES_ have inconsistent lengths.
> 2 (t14a2) ~ >
>
> Perry
>
> On Fri, Jul 26, 2019 at 12:26 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi Perry. Can you please open a new shell and let me know if that
error
> > still occurs if you type "module list"? Thanks!
> >
> > Julie
> >
> > On Fri, Jul 26, 2019 at 10:16 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >
> > > I have this in my .bashrc:
> > >
> > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > > module load met/8.1
> > >
> > > Yet I seem to be stuck on 8.1.
> > >
> > > Plus I am getting this error:
> > >
> > > 23 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts > module
list
> > > Currently Loaded Modulefiles:
> > > ModuleCmd_List.c(146):FATAL:996: The environment variables
> LOADEDMODULES
> > > and _LMFILES_ have inconsistent lengths.
> > > 24 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts
> > >
> > > So I'm not sure what I'm doing wrong here.
> > >
> > > Perry
> > >
> > > On Fri, Jul 26, 2019 at 12:00 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Perry.
> > > >
> > > > I'm not sure what module you are loading or what machine
you're
> using,
> > > but
> > > > when I run the following, I can see that I am using met-8.1.1.
on
> tide:
> > > >
> > > > [Julie.Prestopnik at t21a2 ~]$ module use
> > > > /global/noscrub/Julie.Prestopnik/modulefiles
> > > >
> > > > [Julie.Prestopnik at t21a2 ~]$ module load met/8.1
> > > >
> > > > [Julie.Prestopnik at t21a2 ~]$ pb2nc --version
> > > >
> > > >
> > > > MET Version: V8.1.1
> > > >
> > > > Repository: https://github.com/NCAR/MET
> > > >
> > > > Last Changed Rev: e4fcc77
> > > >
> > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > > >
> > > >
> > > > Please let me know if you have any questions.
> > > >
> > > > Julie
> > > >
> > > > On Fri, Jul 26, 2019 at 9:47 AM perry.shafran at noaa.gov via RT
<
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > > > >
> > > > > Hi, Julie,
> > > > >
> > > > > Seems like it's 8.1 and not 8.1.1. What's the best way to
access
> > 8.1.1
> > > > if
> > > > > the module is still 8.1?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Perry
> > > > >
> > > > > On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Perry.
> > > > > >
> > > > > > To check the version that you are running on WCOSS, you
can type
> > the
> > > > > tool's
> > > > > > name followed by --version. For example:
> > > > > >
> > > > > > "pb2nc --version"
> > > > > >
> > > > > > That should return:
> > > > > >
> > > > > > MET Version: V8.1.1
> > > > > > Repository: https://github.com/NCAR/MET
> > > > > > Last Changed Rev: e4fcc77
> > > > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > > > > >
> > > > > > if you are using met-8.1.1.
> > > > > >
> > > > > > I hope that helps.
> > > > > >
> > > > > > Julie
> > > > > >
> > > > > > On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> >
> > > > > > >
> > > > > > > Hi, Howard,
> > > > > > >
> > > > > > > I put the data that you need online. Here are the
commands
> that
> > I
> > > > have
> > > > > > > used:
> > > > > > >
> > > > > > > pb2nc prepda.2019072412 prepda.2019072412.nc
> PB2NCConfig_aqmmax
> > > > > -pbfile
> > > > > > > prepda.2019072312
> > > > > > > point_stat oz_DAY1 prepda.2019072412.nc
> > > PointStatConfig_AIRNOW_max
> > > > > -v 4
> > > > > > >
> > > > > > > And here is the error I am getting:
> > > > > > >
> > > > > > > DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > DEBUG 2:
> > > > > > > DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0
climatology
> > mean
> > > > > > levels,
> > > > > > > and 0 climatology standard deviation levels.
> > > > > > > DEBUG 2:
> > > > > > > DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > DEBUG 2:
> > > > > > > DEBUG 2: Searching 5110 observations from 5110 messages.
> > > > > > > ERROR :
> > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> error!
> > > > > > > ERROR :
> > > > > > >
> > > > > > > Thanks! It could simply be a case that I have not
properly
> > linked
> > > to
> > > > > the
> > > > > > > fixed version already installed on WCOSS.
> > > > > > >
> > > > > > > Perry
> > > > > > >
> > > > > > > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA
Affiliate
> <
> > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > >
> > > > > > > > Hi, Howard,
> > > > > > > >
> > > > > > > > Do you have access to WCOSS?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Perry
> > > > > > > >
> > > > > > > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Would you upload the BURF file and PB2NC config files
to the
> > ftp
> > > > > site
> > > > > > or
> > > > > > > >> let us know the location we can get?
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >> Howard
> > > > > > > >>
> > > > > > > >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov
wrote:
> > > > > > > >> > I think I am still getting this error, even in
8.1.1.
> > Unless
> > > I
> > > > am
> > > > > > not
> > > > > > > >> > pointing to MET 8.1.1 correctly, but I do believe I
set
> the
> > > > module
> > > > > > to
> > > > > > > >> > 8.1.
> > > > > > > >> >
> > > > > > > >> > Also the interesting thing is that it seems to work
fine
> for
> > > the
> > > > > > > >> > OZMAX1 but
> > > > > > > >> > seems to be choking on the OZMAX8 (the 8-hr
average). I'm
> > > > > wondering
> > > > > > > >> > if we
> > > > > > > >> > can take another look at this?
> > > > > > > >> >
> > > > > > > >> > Thanks!
> > > > > > > >> >
> > > > > > > >> > Perry
> > > > > > > >> >
> > > > > > > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik
via RT
> > > > > > > >> > <met_help at ucar.edu>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Perry.
> > > > > > > >> > >
> > > > > > > >> > > I have updated the installation with met-8.1.1 on
tide
> and
> > > > luna.
> > > > > > > >> > >
> > > > > > > >> > > Here is a link to our Existing Builds page for
this
> > release,
> > > > so
> > > > > > that
> > > > > > > >> > > you
> > > > > > > >> > > can see how to load, but essentially it is the
same as
> > > before
> > > > > for
> > > > > > > >> > > met-8.1
> > > > > > > >> > > (see tide example below):
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > >
> > > >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > > > > > > >> > > 1-existing-builds-metplus-2-1-installations
> > > > > > > >> > >
> > > > > > > >> > > On tide:
> > > > > > > >> > >
> > > > > > > >> > > module use
/global/noscrub/Julie.Prestopnik/modulefiles
> > > > > > > >> > > module load met/8.1
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Please let us know how it goes.
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Julie
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > > > This issue was fixed in met-8.1.1
> > > (bugfix_1137_pb2nc_hdr_vld
> > > > > > > >> > > > branch).
> > > > > > > >> > > > It's not available yet for you. Julie will
install it
> > over
> > > > the
> > > > > > > next
> > > > > > > >> > > couple
> > > > > > > >> > > > of days.
> > > > > > > >> > > >
> > > > > > > >> > > > Cheers,
> > > > > > > >> > > > Howard
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Jul 26 11:08:01 2019
Hmm - interesting. I removed all other references to ics and python
that
already existed, so I'm not entirely sure why this was happening. I
might
have to go to the helpdesk and see what the problem might be. That
might
clear things up to access 8.1.1.
Thanks!
Perry
On Fri, Jul 26, 2019 at 12:36 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Perry.
>
> I am no modulefile expert, but I believe there is perhaps a conflict
the in
> the loaded modules. It could be a direct conflict with something
specified
> in your .bashrc file or it could be something you're loading in your
> .bashrc file and then something that MET or another module is
loading. For
> example, the MET modulefile located
> at /global/noscrub/Julie.Prestopnik/modulefiles/met/8.1 loads:
>
> module load ics/15.0.3
> module use /usrx/local/dev/modulefiles
> module load python/2.7.14
>
> Perhaps there is a problem with one of those? Usually, I get a
message
> that says that one module is being swapped with another module, so
I'm not
> sure why you're getting that error. If you're not able to determine
the
> problem, I suggest talking with one of your IT folks who would
likely know
> more about that than I. I hope that helps!
>
> Julie
>
> On Fri, Jul 26, 2019 at 10:28 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > Yes, it does.
> >
> > 29 (t14a2) ~ > sh
> > 1 (t14a2) ~ > module list
> > Currently Loaded Modulefiles:
> > ModuleCmd_List.c(146):FATAL:996: The environment variables
LOADEDMODULES
> > and _LMFILES_ have inconsistent lengths.
> > 2 (t14a2) ~ >
> >
> > Perry
> >
> > On Fri, Jul 26, 2019 at 12:26 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Hi Perry. Can you please open a new shell and let me know if
that
> error
> > > still occurs if you type "module list"? Thanks!
> > >
> > > Julie
> > >
> > > On Fri, Jul 26, 2019 at 10:16 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > I have this in my .bashrc:
> > > >
> > > > module use /global/noscrub/Julie.Prestopnik/modulefiles
> > > > module load met/8.1
> > > >
> > > > Yet I seem to be stuck on 8.1.
> > > >
> > > > Plus I am getting this error:
> > > >
> > > > 23 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts >
module list
> > > > Currently Loaded Modulefiles:
> > > > ModuleCmd_List.c(146):FATAL:996: The environment variables
> > LOADEDMODULES
> > > > and _LMFILES_ have inconsistent lengths.
> > > > 24 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts
> > > >
> > > > So I'm not sure what I'm doing wrong here.
> > > >
> > > > Perry
> > > >
> > > > On Fri, Jul 26, 2019 at 12:00 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Perry.
> > > > >
> > > > > I'm not sure what module you are loading or what machine
you're
> > using,
> > > > but
> > > > > when I run the following, I can see that I am using met-
8.1.1. on
> > tide:
> > > > >
> > > > > [Julie.Prestopnik at t21a2 ~]$ module use
> > > > > /global/noscrub/Julie.Prestopnik/modulefiles
> > > > >
> > > > > [Julie.Prestopnik at t21a2 ~]$ module load met/8.1
> > > > >
> > > > > [Julie.Prestopnik at t21a2 ~]$ pb2nc --version
> > > > >
> > > > >
> > > > > MET Version: V8.1.1
> > > > >
> > > > > Repository: https://github.com/NCAR/MET
> > > > >
> > > > > Last Changed Rev: e4fcc77
> > > > >
> > > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > > > >
> > > > >
> > > > > Please let me know if you have any questions.
> > > > >
> > > > > Julie
> > > > >
> > > > > On Fri, Jul 26, 2019 at 9:47 AM perry.shafran at noaa.gov via
RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > > > > >
> > > > > > Hi, Julie,
> > > > > >
> > > > > > Seems like it's 8.1 and not 8.1.1. What's the best way to
access
> > > 8.1.1
> > > > > if
> > > > > > the module is still 8.1?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Perry
> > > > > >
> > > > > > On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT <
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Perry.
> > > > > > >
> > > > > > > To check the version that you are running on WCOSS, you
can
> type
> > > the
> > > > > > tool's
> > > > > > > name followed by --version. For example:
> > > > > > >
> > > > > > > "pb2nc --version"
> > > > > > >
> > > > > > > That should return:
> > > > > > >
> > > > > > > MET Version: V8.1.1
> > > > > > > Repository: https://github.com/NCAR/MET
> > > > > > > Last Changed Rev: e4fcc77
> > > > > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
> > > > > > >
> > > > > > > if you are using met-8.1.1.
> > > > > > >
> > > > > > > I hope that helps.
> > > > > > >
> > > > > > > Julie
> > > > > > >
> > > > > > > On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov
via RT
> <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> > >
> > > > > > > >
> > > > > > > > Hi, Howard,
> > > > > > > >
> > > > > > > > I put the data that you need online. Here are the
commands
> > that
> > > I
> > > > > have
> > > > > > > > used:
> > > > > > > >
> > > > > > > > pb2nc prepda.2019072412 prepda.2019072412.nc
> > PB2NCConfig_aqmmax
> > > > > > -pbfile
> > > > > > > > prepda.2019072312
> > > > > > > > point_stat oz_DAY1 prepda.2019072412.nc
> > > > PointStatConfig_AIRNOW_max
> > > > > > -v 4
> > > > > > > >
> > > > > > > > And here is the error I am getting:
> > > > > > > >
> > > > > > > > DEBUG 2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > > DEBUG 2:
> > > > > > > > DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0
climatology
> > > mean
> > > > > > > levels,
> > > > > > > > and 0 climatology standard deviation levels.
> > > > > > > > DEBUG 2:
> > > > > > > > DEBUG 2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > > DEBUG 2:
> > > > > > > > DEBUG 2: Searching 5110 observations from 5110
messages.
> > > > > > > > ERROR :
> > > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> > error!
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > Thanks! It could simply be a case that I have not
properly
> > > linked
> > > > to
> > > > > > the
> > > > > > > > fixed version already installed on WCOSS.
> > > > > > > >
> > > > > > > > Perry
> > > > > > > >
> > > > > > > > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA
> Affiliate
> > <
> > > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > > >
> > > > > > > > > Hi, Howard,
> > > > > > > > >
> > > > > > > > > Do you have access to WCOSS?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > Perry
> > > > > > > > >
> > > > > > > > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Would you upload the BURF file and PB2NC config
files to
> the
> > > ftp
> > > > > > site
> > > > > > > or
> > > > > > > > >> let us know the location we can get?
> > > > > > > > >>
> > > > > > > > >> Cheers,
> > > > > > > > >> Howard
> > > > > > > > >>
> > > > > > > > >> On Thu Jul 25 11:27:38 2019, perry.shafran at noaa.gov
> wrote:
> > > > > > > > >> > I think I am still getting this error, even in
8.1.1.
> > > Unless
> > > > I
> > > > > am
> > > > > > > not
> > > > > > > > >> > pointing to MET 8.1.1 correctly, but I do believe
I set
> > the
> > > > > module
> > > > > > > to
> > > > > > > > >> > 8.1.
> > > > > > > > >> >
> > > > > > > > >> > Also the interesting thing is that it seems to
work fine
> > for
> > > > the
> > > > > > > > >> > OZMAX1 but
> > > > > > > > >> > seems to be choking on the OZMAX8 (the 8-hr
average).
> I'm
> > > > > > wondering
> > > > > > > > >> > if we
> > > > > > > > >> > can take another look at this?
> > > > > > > > >> >
> > > > > > > > >> > Thanks!
> > > > > > > > >> >
> > > > > > > > >> > Perry
> > > > > > > > >> >
> > > > > > > > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie Prestopnik
via RT
> > > > > > > > >> > <met_help at ucar.edu>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Perry.
> > > > > > > > >> > >
> > > > > > > > >> > > I have updated the installation with met-8.1.1
on tide
> > and
> > > > > luna.
> > > > > > > > >> > >
> > > > > > > > >> > > Here is a link to our Existing Builds page for
this
> > > release,
> > > > > so
> > > > > > > that
> > > > > > > > >> > > you
> > > > > > > > >> > > can see how to load, but essentially it is the
same as
> > > > before
> > > > > > for
> > > > > > > > >> > > met-8.1
> > > > > > > > >> > > (see tide example below):
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > >
> > > > >
> > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
> > > > > > > > >> > > 1-existing-builds-metplus-2-1-installations
> > > > > > > > >> > >
> > > > > > > > >> > > On tide:
> > > > > > > > >> > >
> > > > > > > > >> > > module use
> /global/noscrub/Julie.Prestopnik/modulefiles
> > > > > > > > >> > > module load met/8.1
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Please let us know how it goes.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Julie
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > >> > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> > > > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > This issue was fixed in met-8.1.1
> > > > (bugfix_1137_pb2nc_hdr_vld
> > > > > > > > >> > > > branch).
> > > > > > > > >> > > > It's not available yet for you. Julie will
install
> it
> > > over
> > > > > the
> > > > > > > > next
> > > > > > > > >> > > couple
> > > > > > > > >> > > > of days.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Cheers,
> > > > > > > > >> > > > Howard
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Mon Jul 29 07:51:34 2019
Hi, all,
With the help of the WCOSS helpdesk, I found that the source of the
problem
is that I had things set in *both* my .bashrc and my .bash_profile
that
were clobbering each other and causing conflicts. Once I cleared
those
conflicts, I was able to properly access 8.1.1, and thus the problem
in my
AQ verification is solved. The error I had in 8.0 and 8.1 is now gone
and
I am getting a .stat file.
I think if there is still a ticket here we can close it, but I'll let
folks
know about other problems I encounter.
Thanks!
Perry
On Fri, Jul 26, 2019 at 1:07 PM Perry Shafran - NOAA Affiliate <
perry.shafran at noaa.gov> wrote:
> Hmm - interesting. I removed all other references to ics and python
that
> already existed, so I'm not entirely sure why this was happening. I
might
> have to go to the helpdesk and see what the problem might be. That
might
> clear things up to access 8.1.1.
>
> Thanks!
>
> Perry
>
> On Fri, Jul 26, 2019 at 12:36 PM Julie Prestopnik via RT <
> met_help at ucar.edu> wrote:
>
>> Hi Perry.
>>
>> I am no modulefile expert, but I believe there is perhaps a
conflict the
>> in
>> the loaded modules. It could be a direct conflict with something
>> specified
>> in your .bashrc file or it could be something you're loading in
your
>> .bashrc file and then something that MET or another module is
loading.
>> For
>> example, the MET modulefile located
>> at /global/noscrub/Julie.Prestopnik/modulefiles/met/8.1 loads:
>>
>> module load ics/15.0.3
>> module use /usrx/local/dev/modulefiles
>> module load python/2.7.14
>>
>> Perhaps there is a problem with one of those? Usually, I get a
message
>> that says that one module is being swapped with another module, so
I'm not
>> sure why you're getting that error. If you're not able to
determine the
>> problem, I suggest talking with one of your IT folks who would
likely know
>> more about that than I. I hope that helps!
>>
>> Julie
>>
>> On Fri, Jul 26, 2019 at 10:28 AM perry.shafran at noaa.gov via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>> >
>> > Yes, it does.
>> >
>> > 29 (t14a2) ~ > sh
>> > 1 (t14a2) ~ > module list
>> > Currently Loaded Modulefiles:
>> > ModuleCmd_List.c(146):FATAL:996: The environment variables
LOADEDMODULES
>> > and _LMFILES_ have inconsistent lengths.
>> > 2 (t14a2) ~ >
>> >
>> > Perry
>> >
>> > On Fri, Jul 26, 2019 at 12:26 PM Julie Prestopnik via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Hi Perry. Can you please open a new shell and let me know if
that
>> error
>> > > still occurs if you type "module list"? Thanks!
>> > >
>> > > Julie
>> > >
>> > > On Fri, Jul 26, 2019 at 10:16 AM perry.shafran at noaa.gov via RT
<
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
>> > > >
>> > > > I have this in my .bashrc:
>> > > >
>> > > > module use /global/noscrub/Julie.Prestopnik/modulefiles
>> > > > module load met/8.1
>> > > >
>> > > > Yet I seem to be stuck on 8.1.
>> > > >
>> > > > Plus I am getting this error:
>> > > >
>> > > > 23 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts >
module
>> list
>> > > > Currently Loaded Modulefiles:
>> > > > ModuleCmd_List.c(146):FATAL:996: The environment variables
>> > LOADEDMODULES
>> > > > and _LMFILES_ have inconsistent lengths.
>> > > > 24 (t14a2) /meso/save/Perry.Shafran/METplus-2.1/scripts
>> > > >
>> > > > So I'm not sure what I'm doing wrong here.
>> > > >
>> > > > Perry
>> > > >
>> > > > On Fri, Jul 26, 2019 at 12:00 PM Julie Prestopnik via RT <
>> > > > met_help at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Hi Perry.
>> > > > >
>> > > > > I'm not sure what module you are loading or what machine
you're
>> > using,
>> > > > but
>> > > > > when I run the following, I can see that I am using met-
8.1.1. on
>> > tide:
>> > > > >
>> > > > > [Julie.Prestopnik at t21a2 ~]$ module use
>> > > > > /global/noscrub/Julie.Prestopnik/modulefiles
>> > > > >
>> > > > > [Julie.Prestopnik at t21a2 ~]$ module load met/8.1
>> > > > >
>> > > > > [Julie.Prestopnik at t21a2 ~]$ pb2nc --version
>> > > > >
>> > > > >
>> > > > > MET Version: V8.1.1
>> > > > >
>> > > > > Repository: https://github.com/NCAR/MET
>> > > > >
>> > > > > Last Changed Rev: e4fcc77
>> > > > >
>> > > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
>> > > > >
>> > > > >
>> > > > > Please let me know if you have any questions.
>> > > > >
>> > > > > Julie
>> > > > >
>> > > > > On Fri, Jul 26, 2019 at 9:47 AM perry.shafran at noaa.gov via
RT <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>> > > > > >
>> > > > > > Hi, Julie,
>> > > > > >
>> > > > > > Seems like it's 8.1 and not 8.1.1. What's the best way
to
>> access
>> > > 8.1.1
>> > > > > if
>> > > > > > the module is still 8.1?
>> > > > > >
>> > > > > > Thanks!
>> > > > > >
>> > > > > > Perry
>> > > > > >
>> > > > > > On Fri, Jul 26, 2019 at 10:41 AM Julie Prestopnik via RT
<
>> > > > > > met_help at ucar.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Perry.
>> > > > > > >
>> > > > > > > To check the version that you are running on WCOSS, you
can
>> type
>> > > the
>> > > > > > tool's
>> > > > > > > name followed by --version. For example:
>> > > > > > >
>> > > > > > > "pb2nc --version"
>> > > > > > >
>> > > > > > > That should return:
>> > > > > > >
>> > > > > > > MET Version: V8.1.1
>> > > > > > > Repository: https://github.com/NCAR/MET
>> > > > > > > Last Changed Rev: e4fcc77
>> > > > > > > Last Changed Date: Tue Jul 2 17:32:51 2019 -0600
>> > > > > > >
>> > > > > > > if you are using met-8.1.1.
>> > > > > > >
>> > > > > > > I hope that helps.
>> > > > > > >
>> > > > > > > Julie
>> > > > > > >
>> > > > > > > On Fri, Jul 26, 2019 at 7:30 AM perry.shafran at noaa.gov
via
>> RT <
>> > > > > > > met_help at ucar.edu> wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>> > >
>> > > > > > > >
>> > > > > > > > Hi, Howard,
>> > > > > > > >
>> > > > > > > > I put the data that you need online. Here are the
commands
>> > that
>> > > I
>> > > > > have
>> > > > > > > > used:
>> > > > > > > >
>> > > > > > > > pb2nc prepda.2019072412 prepda.2019072412.nc
>> > PB2NCConfig_aqmmax
>> > > > > > -pbfile
>> > > > > > > > prepda.2019072312
>> > > > > > > > point_stat oz_DAY1 prepda.2019072412.nc
>> > > > PointStatConfig_AIRNOW_max
>> > > > > > -v 4
>> > > > > > > >
>> > > > > > > > And here is the error I am getting:
>> > > > > > > >
>> > > > > > > > DEBUG 2:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------
>> > > > > > > > DEBUG 2:
>> > > > > > > > DEBUG 2: Reading data for OZMAX8/L1.
>> > > > > > > > DEBUG 2: For OZMAX8/L1 found 1 forecast levels, 0
>> climatology
>> > > mean
>> > > > > > > levels,
>> > > > > > > > and 0 climatology standard deviation levels.
>> > > > > > > > DEBUG 2:
>> > > > > > > > DEBUG 2:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------
>> > > > > > > > DEBUG 2:
>> > > > > > > > DEBUG 2: Searching 5110 observations from 5110
messages.
>> > > > > > > > ERROR :
>> > > > > > > > ERROR : StringArray::operator[](int) const -> range
check
>> > error!
>> > > > > > > > ERROR :
>> > > > > > > >
>> > > > > > > > Thanks! It could simply be a case that I have not
properly
>> > > linked
>> > > > to
>> > > > > > the
>> > > > > > > > fixed version already installed on WCOSS.
>> > > > > > > >
>> > > > > > > > Perry
>> > > > > > > >
>> > > > > > > > On Thu, Jul 25, 2019 at 4:32 PM Perry Shafran - NOAA
>> Affiliate
>> > <
>> > > > > > > > perry.shafran at noaa.gov> wrote:
>> > > > > > > >
>> > > > > > > > > Hi, Howard,
>> > > > > > > > >
>> > > > > > > > > Do you have access to WCOSS?
>> > > > > > > > >
>> > > > > > > > > Thanks!
>> > > > > > > > >
>> > > > > > > > > Perry
>> > > > > > > > >
>> > > > > > > > > On Thu, Jul 25, 2019 at 3:34 PM Howard Soh via RT <
>> > > > > met_help at ucar.edu
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Would you upload the BURF file and PB2NC config
files to
>> the
>> > > ftp
>> > > > > > site
>> > > > > > > or
>> > > > > > > > >> let us know the location we can get?
>> > > > > > > > >>
>> > > > > > > > >> Cheers,
>> > > > > > > > >> Howard
>> > > > > > > > >>
>> > > > > > > > >> On Thu Jul 25 11:27:38 2019,
perry.shafran at noaa.gov
>> wrote:
>> > > > > > > > >> > I think I am still getting this error, even in
8.1.1.
>> > > Unless
>> > > > I
>> > > > > am
>> > > > > > > not
>> > > > > > > > >> > pointing to MET 8.1.1 correctly, but I do
believe I set
>> > the
>> > > > > module
>> > > > > > > to
>> > > > > > > > >> > 8.1.
>> > > > > > > > >> >
>> > > > > > > > >> > Also the interesting thing is that it seems to
work
>> fine
>> > for
>> > > > the
>> > > > > > > > >> > OZMAX1 but
>> > > > > > > > >> > seems to be choking on the OZMAX8 (the 8-hr
average).
>> I'm
>> > > > > > wondering
>> > > > > > > > >> > if we
>> > > > > > > > >> > can take another look at this?
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks!
>> > > > > > > > >> >
>> > > > > > > > >> > Perry
>> > > > > > > > >> >
>> > > > > > > > >> > On Wed, Jul 10, 2019 at 12:04 PM Julie
Prestopnik via
>> RT
>> > > > > > > > >> > <met_help at ucar.edu>
>> > > > > > > > >> > wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > Hi Perry.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I have updated the installation with met-8.1.1
on
>> tide
>> > and
>> > > > > luna.
>> > > > > > > > >> > >
>> > > > > > > > >> > > Here is a link to our Existing Builds page for
this
>> > > release,
>> > > > > so
>> > > > > > > that
>> > > > > > > > >> > > you
>> > > > > > > > >> > > can see how to load, but essentially it is the
same
>> as
>> > > > before
>> > > > > > for
>> > > > > > > > >> > > met-8.1
>> > > > > > > > >> > > (see tide example below):
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > >
>> > > > >
>> > https://dtcenter.org/community-code/model-evaluation-tools-
met/metv8-
>> > > > > > > > >> > > 1-existing-builds-metplus-2-1-installations
>> > > > > > > > >> > >
>> > > > > > > > >> > > On tide:
>> > > > > > > > >> > >
>> > > > > > > > >> > > module use
>> /global/noscrub/Julie.Prestopnik/modulefiles
>> > > > > > > > >> > > module load met/8.1
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > Please let us know how it goes.
>> > > > > > > > >> > >
>> > > > > > > > >> > > Thanks,
>> > > > > > > > >> > > Julie
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Tue, Jul 9, 2019 at 3:57 PM Howard Soh via
RT <
>> > > > > > > met_help at ucar.edu
>> > > > > > > > >
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > <URL:
>> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>> > > > > > >
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > This issue was fixed in met-8.1.1
>> > > > (bugfix_1137_pb2nc_hdr_vld
>> > > > > > > > >> > > > branch).
>> > > > > > > > >> > > > It's not available yet for you. Julie will
install
>> it
>> > > over
>> > > > > the
>> > > > > > > > next
>> > > > > > > > >> > > couple
>> > > > > > > > >> > > > of days.
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Cheers,
>> > > > > > > > >> > > > Howard
>> > > > > > > > >> > > >
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
More information about the Met_help
mailing list