[Met_help] [rt.rap.ucar.edu #89921] History for Re: PB2NC time summary questions
David Fillmore via RT
met_help at ucar.edu
Tue Jul 9 15:28:41 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi, everyone,
I am attaching this to this long thread regarding the daily max of ozone
verification, so we remember what we were doing back then.
I am now running this again using MET 8.0, and something seems to not be
working when I run point_stat. I get this error when running a daily max
of ozone:
ERROR :
ERROR : StringArray::operator[](int) const -> range check error!
ERROR :
I am using my own scripts to run this using MET, so this is not using
METplus.
You can find a log file of this here:
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430. It
runs pb2nc and point_stat for max ozone. The pb2nc code is working fine, I
believe.
The script I am using to run this is here:
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
The prepbufr files are:
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
The model file is here:
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
What could be causing this error?
Thanks!
Perry
On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <johnhg at ucar.edu> wrote:
> Great, glad that did the trick.
>
> John
>
> On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> perry.shafran at noaa.gov> wrote:
>
>> Well this worked! And Jeff confirmed to me that the 8-hr average
>> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr avg.
>>
>> Thanks!
>>
>> Perry
>>
>> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway <johnhg at ucar.edu>
>> wrote:
>>
>>> Perry,
>>>
>>> I think the problem is in your setting of valid_time="${CUR_VX_TIME}";
>>>
>>> Here's your wgrib2 output:
>>> wgrib2 oz_DAY1
>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
>>>
>>> The "d=2018081112" tells you the model initialization time. The valid
>>> time = init time + forecast time. So the OZMAX1 data is valid 16 hours
>>> later while the OZMAX8 data is valid 22 hours later. Rerun that wgrib2
>>> command using the "-vt" argument to list the verification time.
>>>
>>> Point-Stat isn't finding any OZMAX8 data because that data isn't valid
>>> at the time you're requesting.
>>>
>>> John
>>>
>>>
>>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate <
>>> perry.shafran at noaa.gov> wrote:
>>>
>>>> Hi, John,
>>>>
>>>> I haven't been able to work on this for awhile, but now I'm working on
>>>> it again but seem to be running into issues.
>>>>
>>>> I have a file that has both the daily max for 1-hr ozone and the daily
>>>> max for 8-hr ozone. The wgrib2 output looks like this:
>>>>
>>>> 9 (t10a2) /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
>>>> wgrib2 oz_DAY1
>>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
>>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
>>>> 10 (t10a2) /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
>>>>
>>>> As you can see they have different forecast times. But when I run
>>>> point_stat, I get this output:
>>>>
>>>> DEBUG 2:
>>>> DEBUG 2:
>>>> --------------------------------------------------------------------------------
>>>> DEBUG 2:
>>>> DEBUG 2: Reading data for OZMAX8/L1.
>>>> WARNING:
>>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching records
>>>> found for 'OZMAX8/L1'
>>>> WARNING:
>>>> WARNING:
>>>> WARNING: process_fcst_climo_files() -> no fields matching OZMAX8/L1
>>>> found in file: oz_DAY1
>>>> WARNING:
>>>>
>>>> Here is the relevant portion of the point_stat config file:
>>>>
>>>> fcst = {
>>>> field = [
>>>> { name = "OZMAX1"; level = "L1"; valid_time="${CUR_VX_TIME}"; },
>>>> { name = "OZMAX8"; level = "L1"; valid_time="${CUR_VX_TIME}"; }
>>>> ];
>>>> }
>>>>
>>>> obs = {
>>>> convert(x) = x * 10^9;
>>>> field = [
>>>> { name = "COPO"; level = "A1"; message_type =
>>>> "AIRNOW_MAX_230000"; },
>>>> { name = "COPO"; level = "A8"; message_type =
>>>> "AIRNOW_MAX_230000"; }
>>>> ];
>>>> }
>>>>
>>>> I don't remember if I had this working previously for both the 1hr and
>>>> the 8hr ozone, but apparently there is some sort of issue for the 8hr
>>>> ozone. Have any thoughts?
>>>>
>>>> Thanks!
>>>>
>>>> Perry
>>>>
>>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA Affiliate <
>>>> perry.shafran at noaa.gov> wrote:
>>>>
>>>>> I chose option 2, and that seems to work - I have a .stat file with
>>>>> output. Now I'll need to do some work here to see if the output looks OK,
>>>>> and to configure the verification so it verifies OZMAX1 and OZMAX8.
>>>>>
>>>>> I also wonder if this would be easily modified for PM.
>>>>>
>>>>> Perry
>>>>>
>>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway <johnhg at ucar.edu>
>>>>> wrote:
>>>>>
>>>>>> Perry,
>>>>>>
>>>>>> Yep, you've configured/run PB2NC to compute time summaries that are
>>>>>> valid at 04Z. And you're comparing them to forecasts that are purportedly
>>>>>> valid at 05Z. And that's the reason why you have so many obs rejected due
>>>>>> to the valid time.
>>>>>>
>>>>>> It's up to you guys to decide what the "correct" way of handling this
>>>>>> is... but here are a couple of options:
>>>>>>
>>>>>> (1) You could use the existing setup and address this logic in
>>>>>> Point-Stat config file. Change the matching time obs_window from 0 to:
>>>>>>
>>>>>> obs_window = {
>>>>>> beg = 3600;
>>>>>> end = 3600;
>>>>>> }
>>>>>> This tells Point-Stat to match observations which are exactly 1-hour
>>>>>> prior to the valid time of the forecast.
>>>>>>
>>>>>> (2) You could rerun PB2NC to compute a time summary from 05Z to 05Z
>>>>>> instead and then leave the Point-Stat obs_window setting at 0:
>>>>>>
>>>>>> time_summary = {
>>>>>> flag = TRUE;
>>>>>> beg = "050000";
>>>>>> end = "050000";
>>>>>> ...
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA Affiliate <
>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>
>>>>>>> Oops, spoke too soon. There *is* a .stat output file, but it's
>>>>>>> empty. Here's the output from point_stat:
>>>>>>>
>>>>>>> DEBUG 4:
>>>>>>> DEBUG 4: Data plane information:
>>>>>>> DEBUG 4: plane min: 13.86
>>>>>>> DEBUG 4: plane max: 130.82
>>>>>>> DEBUG 4: scan mode: 64
>>>>>>> DEBUG 4: valid time: 20180709_050000
>>>>>>> DEBUG 4: lead time: 170000
>>>>>>> DEBUG 4: init time: 20180708_120000
>>>>>>> DEBUG 4: bitmap flag: 255
>>>>>>>
>>>>>>>
>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for observation type
>>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation method NEAREST(1),
>>>>>>> using 0 pairs.
>>>>>>> DEBUG 3: Number of matched pairs = 0
>>>>>>> DEBUG 3: Observations processed = 124637
>>>>>>> DEBUG 3: Rejected: SID exclusion = 0
>>>>>>> DEBUG 3: Rejected: obs type = 0
>>>>>>> DEBUG 3: Rejected: valid time = 122154
>>>>>>> DEBUG 3: Rejected: bad obs value = 0
>>>>>>> DEBUG 3: Rejected: off the grid = 70
>>>>>>> DEBUG 3: Rejected: level mismatch = 1224
>>>>>>> DEBUG 3: Rejected: quality marker = 0
>>>>>>> DEBUG 3: Rejected: message type = 1189
>>>>>>> DEBUG 3: Rejected: masking region = 0
>>>>>>> DEBUG 3: Rejected: bad fcst value = 0
>>>>>>> DEBUG 3: Rejected: duplicates = 0
>>>>>>>
>>>>>>> Looks like most rejections were due to valid time.
>>>>>>>
>>>>>>> Have any thoughts about this one?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Perry
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA Affiliate <
>>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>>
>>>>>>>> Hi, John,
>>>>>>>>
>>>>>>>> Looks like the changing of the time stamp from _04 to _05 did the
>>>>>>>> trick as point_stat ran without an issue with that change.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Perry
>>>>>>>>
>>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>> Perry,
>>>>>>>>>
>>>>>>>>> I agree, there's likely some issue with requesting the timestamp
>>>>>>>>> of the data from that file. I think switching the verification time to
>>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
>>>>>>>>>
>>>>>>>>> Looks like wgrib2 and MET both agree on the initialization times
>>>>>>>>> and valid times for this data. Here's some useful wgrib2 and MET commands
>>>>>>>>> to illustrate.
>>>>>>>>>
>>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast time
>>>>>>>>> (-ftime), and valid time (-vt). Here it is using a sample data file you
>>>>>>>>> sent to me. But remember I'm still using the "old" wgrib2 so I have that
>>>>>>>>> funky forecast hour:
>>>>>>>>>
>>>>>>>>> wgrib2 -t -ftime -vt aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
>>>>>>>>> fcst:vt=2018070505
>>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
>>>>>>>>>
>>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity level (-v
>>>>>>>>> 4) to have MET tell you the timing info of the data it is reading:
>>>>>>>>>
>>>>>>>>> # Read the first record
>>>>>>>>> plot_data_plane aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
>>>>>>>>> ...
>>>>>>>>> DEBUG 4: valid time: 20180705_050000
>>>>>>>>> DEBUG 4: lead time: 230000
>>>>>>>>> DEBUG 4: init time: 20180704_060000
>>>>>>>>> ...
>>>>>>>>> # Read the second record
>>>>>>>>> plot_data_plane aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
>>>>>>>>> ...
>>>>>>>>> DEBUG 4: valid time: 20180706_050000
>>>>>>>>> DEBUG 4: lead time: 470000
>>>>>>>>> DEBUG 4: init time: 20180704_060000
>>>>>>>>>
>>>>>>>>> Hope that helps.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA Affiliate <
>>>>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, John,
>>>>>>>>>>
>>>>>>>>>> When I implemented all the changes (and also set it up so it's
>>>>>>>>>> reading the beta6 version), I used this command:
>>>>>>>>>>
>>>>>>>>>> CUR_VX_TIME=20180709_04
>>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
>>>>>>>>>> /meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max -v 4
>>>>>>>>>>
>>>>>>>>>> But then I got this error:
>>>>>>>>>>
>>>>>>>>>> DEBUG 2:
>>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
>>>>>>>>>> WARNING:
>>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
>>>>>>>>>> records found for 'OZMAX1/L1'
>>>>>>>>>> WARNING:
>>>>>>>>>> WARNING:
>>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields matching
>>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
>>>>>>>>>> WARNING:
>>>>>>>>>> ERROR :
>>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested forecast data
>>>>>>>>>> found! Exiting...
>>>>>>>>>>
>>>>>>>>>> Not sure why that is. A look at the wgrib2 output for the file
>>>>>>>>>> in this case reads:
>>>>>>>>>>
>>>>>>>>>> 14 (t14a2)
>>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 > wgrib2 oz1hr_DAY1
>>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave fcst:
>>>>>>>>>> 15 (t14a2) /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>>>>>>>>>>
>>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had suggested
>>>>>>>>>> that it should be the time stamp of the 2nd day, but maybe it needs to be
>>>>>>>>>> the time stamp of the first day?
>>>>>>>>>>
>>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
>>>>>>>>>>
>>>>>>>>>> Thanks for any thoughts.
>>>>>>>>>>
>>>>>>>>>> Perry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> Perry,
>>>>>>>>>>>
>>>>>>>>>>> I'm confused. In the previous air quality vx we did, we used
>>>>>>>>>>> observation files named:
>>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
>>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
>>>>>>>>>>>
>>>>>>>>>>> And those files don't exist in the July data you just sent.
>>>>>>>>>>> Instead, it's BUFR files with names like:
>>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d aqm.t12z.airnow.tm024.bufr_d
>>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
>>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure if you're actually verifying against the BUFR or
>>>>>>>>>>> PREPBUFR.
>>>>>>>>>>>
>>>>>>>>>>> Let me just tell you the key points for the config file
>>>>>>>>>>> settings...
>>>>>>>>>>>
>>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input files -
>>>>>>>>>>> thecurrent day and the previous day...
>>>>>>>>>>> pb2nc \
>>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
>>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
>>>>>>>>>>> PB2NCConfig \
>>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>
>>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary like this:
>>>>>>>>>>> time_summary = {
>>>>>>>>>>> flag = TRUE;
>>>>>>>>>>> beg = "040000";
>>>>>>>>>>> end = "040000";
>>>>>>>>>>> step = 24*3600;
>>>>>>>>>>> width = { beg = -23*3600; end = 0; } // previous
>>>>>>>>>>> 23-hours (05Z to 04Z)
>>>>>>>>>>> grib_code = [];
>>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
>>>>>>>>>>> type = [ "max", "mean" ];
>>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
>>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when at least
>>>>>>>>>>> 75% of expect obs are present
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> That'll compute the MAX and MEAN values at each station over the
>>>>>>>>>>> defined time period. They're identified in the output by the message type:
>>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
>>>>>>>>>>>
>>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and "obs"
>>>>>>>>>>> sections like this for the max of the 1-hour data. Instead of specifying
>>>>>>>>>>> "lead_time" to filter the data, it may be more convenient to specify the
>>>>>>>>>>> valid time. So that you don't have to modify the config file, I'd suggest
>>>>>>>>>>> using environment variables:
>>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
>>>>>>>>>>>
>>>>>>>>>>> fcst = {
>>>>>>>>>>> field = [
>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
>>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
>>>>>>>>>>> ];
>>>>>>>>>>> }
>>>>>>>>>>> obs = {
>>>>>>>>>>> convert(x) = x * 10^9;
>>>>>>>>>>> field = [
>>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
>>>>>>>>>>> "AIRNOW_MAX_230000"; }
>>>>>>>>>>> ];
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sounds good, thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA Affiliate <
>>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids and focus on
>>>>>>>>>>>>> the non-bc files. It takes some time to copy files over to Theia so we'll
>>>>>>>>>>>>> let him work on the files he has for now and if he runs into problems, I'll
>>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr max/avg files).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not for bias
>>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe, you may want to provide
>>>>>>>>>>>>>> him para5 runs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for John to do
>>>>>>>>>>>>>> some testing some production files. Do you think I should use parallels
>>>>>>>>>>>>>> instead or do you think the ops files are OK for creating a MET+ use case
>>>>>>>>>>>>>> for AQ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know your thoughts. I think that the ops files should
>>>>>>>>>>>>>> be OK for now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels. We
>>>>>>>>>>>>>>> wanted to update the error in daily max hour labels but management did not
>>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
>>>>>>>>>>>>>>> Jeff
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA Affiliate wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do any of the current parallels have the correct time
>>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John over parallel CMAQ data
>>>>>>>>>>>>>>> rather than production CMAQ.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't think that the MET team has any other versions of
>>>>>>>>>>>>>>> wgrib2 than the one they already have - but they can also clarify this as
>>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does over the current wgrib2?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
>>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to use the
>>>>>>>>>>>>>>>> latest parallel run output (para5).
>>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall Jianping also
>>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management did not want to risk any
>>>>>>>>>>>>>>>> further delays so we will have to do so with the FYQ4 implementation.
>>>>>>>>>>>>>>>> Jeff
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA Affiliate wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I think I
>>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time ago, but I don't know if I
>>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I think the parallel CMAQ they
>>>>>>>>>>>>>>>> are running corrects that, but not too entirely sure. But he would have
>>>>>>>>>>>>>>>> more insight as to what is correct and how to approach this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John talks
>>>>>>>>>>>>>>>> about?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Perry
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley Gotway <
>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Perry,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to compute the
>>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-hour ozone concentrations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I ran pb2nc like this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
>>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00 pb2nc_output.nc
>>>>>>>>>>>>>>>>> PB2NCConfig \
>>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And then I ran point_stat like this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat \
>>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
>>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
>>>>>>>>>>>>>>>>> -outdir point_stat -v 4
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Using the attached config files.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
>>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
>>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region FULL, for interpolation
>>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But I noticed some oddities:
>>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of this "max"
>>>>>>>>>>>>>>>>> file:
>>>>>>>>>>>>>>>>> wgrib2 aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
>>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
>>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
>>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-40 hour
>>>>>>>>>>>>>>>>> ave fcst:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing revealed
>>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours 17 and 41, respectively.
>>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the same file, we
>>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which one to use. Do this by
>>>>>>>>>>>>>>>>> defining the "lead_time":
>>>>>>>>>>>>>>>>> fcst = {
>>>>>>>>>>>>>>>>> field = [
>>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1"; lead_time =
>>>>>>>>>>>>>>>>> "41";' }
>>>>>>>>>>>>>>>>> ];
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are off by 1
>>>>>>>>>>>>>>>>> hour.
>>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead time =
>>>>>>>>>>>>>>>>> valid time of 2018052805.
>>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max value from
>>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z while the obs are valid at 04Z.
>>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat config file by
>>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
>>>>>>>>>>>>>>>>> obs_window = {
>>>>>>>>>>>>>>>>> beg = -3600;
>>>>>>>>>>>>>>>>> end = -3600;
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So what're the next steps?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh <hsoh at ucar.edu>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We're making progress on the time summaries, but I found
>>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota here:
>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute 24-hour
>>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the output to an ascii file using
>>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to count up the number of regular
>>>>>>>>>>>>>>>>>> and maximum obs values:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt | wc -l
>>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt | sort -u
>>>>>>>>>>>>>>>>>> | wc -l
>>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt | wc -l
>>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt | sort -u |
>>>>>>>>>>>>>>>>>> wc -l
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that all of
>>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555 meaning
>>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are duplicates.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For example, the last station ID in that file is named
>>>>>>>>>>>>>>>>>> "2FB31EEC":
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep 2FB31EEC
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47 -114.77
>>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in the last two
>>>>>>>>>>>>>>>>>> lines.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Could you please check to see why these duplicates are
>>>>>>>>>>>>>>>>>> showing up?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh <hsoh at ucar.edu>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> No, I don't have them yet.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers.
>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why we'd have
>>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across 2 files. Do we have existing
>>>>>>>>>>>>>>>>>>> examples of smaller files from which we could compute a 24-hour max
>>>>>>>>>>>>>>>>>>> temperature or something?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes (550.319 sec).
>>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
>>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in the new
>>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh <hsoh at ucar.edu>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content of obs_hid
>>>>>>>>>>>>>>>>>>>>> variable).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary (test
>>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
>>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I added one
>>>>>>>>>>>>>>>>>>>>> with time summary and two input files and will check the unit test today.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change the nightly
>>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently covered? If not, should I add
>>>>>>>>>>>>>>>>>>>>> it as a new unit test?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary data.
>>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The output is available
>>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
>>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley Gotway <
>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic for PB2NC.
>>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-hour maximum values of COPO from
>>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr counts:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to dump to ascii
>>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000 string show up in the output.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC not actually
>>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the Rscript dumping the data.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
>>>>>>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John Halley Gotway <
>>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard Soh <
>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
>>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and 57246 observations
>>>>>>>>>>>>>>>>>>>>>>>>> from aqm.20180528/aqm.t12z.prepbufr.tm00. There
>>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data. I'm working on this.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John Halley Gotway
>>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please take a
>>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC. I copied the data over to
>>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
>>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
>>>>>>>>>>>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
>>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly maximums
>>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
>>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
>>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
>>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
>>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
>>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end = 0; } //
>>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
>>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
>>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
>>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
>>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
>>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but using
>>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what you'd expect. The hdr_typ_table
>>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the "AIRNOW_MAX_240000" message type, but as you pointed
>>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused by using 2
>>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them to cover the desired 04Z to 04Z
>>>>>>>>>>>>>>>>>>>>>>>>>> time window.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John Halley
>>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW PrepBUFR file
>>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24 hour maximum of obs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
>>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is reasonable for me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data in NetCDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of "AIRNOW_MAX_240000".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was generated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without any
>>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers than obs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF output:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295 currently)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to 3655.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for summary) exists
>>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable. The first header index for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than 3655.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
>>>>>>>>>>>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ndas.20120409.t12z.prepbufr.tm00.summary.nc
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1 -d" " | sort
>>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley Gotway wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota in here:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180622
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to ascii:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript /d1/johnhg/MET/MET_development/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
>>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all of the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of them should actually be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by this:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table pb2nc_output.nc
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table lookup
>>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Fri Apr 26 10:08:38 2019
Hi Perry -
I'm also forwarding this ticket to Howard,
I think John is quite busy with the MET 8.1 release so it might be
later
next week before he gets a chance to look at this.
If you want to upload your data and config files to our ftp server we
can
replicate and debug the error here.
I can have a first look this afternoon.
You can create a subdir at
ftp.rap.ucar.edu://incoming/irap/met_help
thanks,
David
On Thu, Apr 25, 2019 at 11:17 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Thu Apr 25 11:16:42 2019: Request 89921 was acted upon.
> Transaction: Ticket created by perry.shafran at noaa.gov
> Queue: met_help
> Subject: Re: PB2NC time summary questions
> Owner: Nobody
> Requestors: perry.shafran at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
>
> Hi, everyone,
>
> I am attaching this to this long thread regarding the daily max of
ozone
> verification, so we remember what we were doing back then.
>
> I am now running this again using MET 8.0, and something seems to
not be
> working when I run point_stat. I get this error when running a
daily max
> of ozone:
>
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> I am using my own scripts to run this using MET, so this is not
using
> METplus.
>
> You can find a log file of this here:
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
It
> runs pb2nc and point_stat for max ozone. The pb2nc code is working
fine, I
> believe.
>
> The script I am using to run this is here:
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
>
> The prepbufr files are:
>
> /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
>
> The model file is here:
>
>
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
>
>
> What could be causing this error?
>
> Thanks!
>
> Perry
>
>
> On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Great, glad that did the trick.
> >
> > John
> >
> > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > perry.shafran at noaa.gov> wrote:
> >
> >> Well this worked! And Jeff confirmed to me that the 8-hr average
> >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
avg.
> >>
> >> Thanks!
> >>
> >> Perry
> >>
> >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
<johnhg at ucar.edu>
> >> wrote:
> >>
> >>> Perry,
> >>>
> >>> I think the problem is in your setting of
valid_time="${CUR_VX_TIME}";
> >>>
> >>> Here's your wgrib2 output:
> >>> wgrib2 oz_DAY1
> >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> >>>
> >>> The "d=2018081112" tells you the model initialization time. The
valid
> >>> time = init time + forecast time. So the OZMAX1 data is valid
16 hours
> >>> later while the OZMAX8 data is valid 22 hours later. Rerun that
wgrib2
> >>> command using the "-vt" argument to list the verification time.
> >>>
> >>> Point-Stat isn't finding any OZMAX8 data because that data isn't
valid
> >>> at the time you're requesting.
> >>>
> >>> John
> >>>
> >>>
> >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate <
> >>> perry.shafran at noaa.gov> wrote:
> >>>
> >>>> Hi, John,
> >>>>
> >>>> I haven't been able to work on this for awhile, but now I'm
working on
> >>>> it again but seem to be running into issues.
> >>>>
> >>>> I have a file that has both the daily max for 1-hr ozone and
the daily
> >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> >>>>
> >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> >>>> wgrib2 oz_DAY1
> >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
> >>>> 10 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> >>>>
> >>>> As you can see they have different forecast times. But when I
run
> >>>> point_stat, I get this output:
> >>>>
> >>>> DEBUG 2:
> >>>> DEBUG 2:
> >>>>
>
--------------------------------------------------------------------------------
> >>>> DEBUG 2:
> >>>> DEBUG 2: Reading data for OZMAX8/L1.
> >>>> WARNING:
> >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
records
> >>>> found for 'OZMAX8/L1'
> >>>> WARNING:
> >>>> WARNING:
> >>>> WARNING: process_fcst_climo_files() -> no fields matching
OZMAX8/L1
> >>>> found in file: oz_DAY1
> >>>> WARNING:
> >>>>
> >>>> Here is the relevant portion of the point_stat config file:
> >>>>
> >>>> fcst = {
> >>>> field = [
> >>>> { name = "OZMAX1"; level = "L1";
valid_time="${CUR_VX_TIME}"; },
> >>>> { name = "OZMAX8"; level = "L1";
valid_time="${CUR_VX_TIME}"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> obs = {
> >>>> convert(x) = x * 10^9;
> >>>> field = [
> >>>> { name = "COPO"; level = "A1"; message_type =
> >>>> "AIRNOW_MAX_230000"; },
> >>>> { name = "COPO"; level = "A8"; message_type =
> >>>> "AIRNOW_MAX_230000"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> I don't remember if I had this working previously for both the
1hr and
> >>>> the 8hr ozone, but apparently there is some sort of issue for
the 8hr
> >>>> ozone. Have any thoughts?
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Perry
> >>>>
> >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA Affiliate
<
> >>>> perry.shafran at noaa.gov> wrote:
> >>>>
> >>>>> I chose option 2, and that seems to work - I have a .stat file
with
> >>>>> output. Now I'll need to do some work here to see if the
output
> looks OK,
> >>>>> and to configure the verification so it verifies OZMAX1 and
OZMAX8.
> >>>>>
> >>>>> I also wonder if this would be easily modified for PM.
> >>>>>
> >>>>> Perry
> >>>>>
> >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
<johnhg at ucar.edu
> >
> >>>>> wrote:
> >>>>>
> >>>>>> Perry,
> >>>>>>
> >>>>>> Yep, you've configured/run PB2NC to compute time summaries
that are
> >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> purportedly
> >>>>>> valid at 05Z. And that's the reason why you have so many obs
> rejected due
> >>>>>> to the valid time.
> >>>>>>
> >>>>>> It's up to you guys to decide what the "correct" way of
handling
> this
> >>>>>> is... but here are a couple of options:
> >>>>>>
> >>>>>> (1) You could use the existing setup and address this logic
in
> >>>>>> Point-Stat config file. Change the matching time obs_window
from 0
> to:
> >>>>>>
> >>>>>> obs_window = {
> >>>>>> beg = 3600;
> >>>>>> end = 3600;
> >>>>>> }
> >>>>>> This tells Point-Stat to match observations which are exactly
1-hour
> >>>>>> prior to the valid time of the forecast.
> >>>>>>
> >>>>>> (2) You could rerun PB2NC to compute a time summary from 05Z
to 05Z
> >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> >>>>>>
> >>>>>> time_summary = {
> >>>>>> flag = TRUE;
> >>>>>> beg = "050000";
> >>>>>> end = "050000";
> >>>>>> ...
> >>>>>>
> >>>>>> Thanks,
> >>>>>> John
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate <
> >>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>
> >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
it's
> >>>>>>> empty. Here's the output from point_stat:
> >>>>>>>
> >>>>>>> DEBUG 4:
> >>>>>>> DEBUG 4: Data plane information:
> >>>>>>> DEBUG 4: plane min: 13.86
> >>>>>>> DEBUG 4: plane max: 130.82
> >>>>>>> DEBUG 4: scan mode: 64
> >>>>>>> DEBUG 4: valid time: 20180709_050000
> >>>>>>> DEBUG 4: lead time: 170000
> >>>>>>> DEBUG 4: init time: 20180708_120000
> >>>>>>> DEBUG 4: bitmap flag: 255
> >>>>>>>
> >>>>>>>
> >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation type
> >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> NEAREST(1),
> >>>>>>> using 0 pairs.
> >>>>>>> DEBUG 3: Number of matched pairs = 0
> >>>>>>> DEBUG 3: Observations processed = 124637
> >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> >>>>>>> DEBUG 3: Rejected: obs type = 0
> >>>>>>> DEBUG 3: Rejected: valid time = 122154
> >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> >>>>>>> DEBUG 3: Rejected: off the grid = 70
> >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> >>>>>>> DEBUG 3: Rejected: quality marker = 0
> >>>>>>> DEBUG 3: Rejected: message type = 1189
> >>>>>>> DEBUG 3: Rejected: masking region = 0
> >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> >>>>>>> DEBUG 3: Rejected: duplicates = 0
> >>>>>>>
> >>>>>>> Looks like most rejections were due to valid time.
> >>>>>>>
> >>>>>>> Have any thoughts about this one?
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> Perry
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
Affiliate <
> >>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>
> >>>>>>>> Hi, John,
> >>>>>>>>
> >>>>>>>> Looks like the changing of the time stamp from _04 to _05
did the
> >>>>>>>> trick as point_stat ran without an issue with that change.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Perry
> >>>>>>>>
> >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> >>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>
> >>>>>>>>> Perry,
> >>>>>>>>>
> >>>>>>>>> I agree, there's likely some issue with requesting the
timestamp
> >>>>>>>>> of the data from that file. I think switching the
verification
> time to
> >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> >>>>>>>>>
> >>>>>>>>> Looks like wgrib2 and MET both agree on the initialization
times
> >>>>>>>>> and valid times for this data. Here's some useful wgrib2
and
> MET commands
> >>>>>>>>> to illustrate.
> >>>>>>>>>
> >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
time
> >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a sample
data
> file you
> >>>>>>>>> sent to me. But remember I'm still using the "old" wgrib2
so I
> have that
> >>>>>>>>> funky forecast hour:
> >>>>>>>>>
> >>>>>>>>> wgrib2 -t -ftime -vt
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> >>>>>>>>> fcst:vt=2018070505
> >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
> >>>>>>>>>
> >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level (-v
> >>>>>>>>> 4) to have MET tell you the timing info of the data it is
> reading:
> >>>>>>>>>
> >>>>>>>>> # Read the first record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> >>>>>>>>> DEBUG 4: lead time: 230000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>> ...
> >>>>>>>>> # Read the second record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> >>>>>>>>> DEBUG 4: lead time: 470000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>>
> >>>>>>>>> Hope that helps.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> John
> >>>>>>>>>
> >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
Affiliate <
> >>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi, John,
> >>>>>>>>>>
> >>>>>>>>>> When I implemented all the changes (and also set it up so
it's
> >>>>>>>>>> reading the beta6 version), I used this command:
> >>>>>>>>>>
> >>>>>>>>>> CUR_VX_TIME=20180709_04
> >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> >>>>>>>>>>
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
-v 4
> >>>>>>>>>>
> >>>>>>>>>> But then I got this error:
> >>>>>>>>>>
> >>>>>>>>>> DEBUG 2:
> >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> >>>>>>>>>> records found for 'OZMAX1/L1'
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields matching
> >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> >>>>>>>>>> WARNING:
> >>>>>>>>>> ERROR :
> >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> data
> >>>>>>>>>> found! Exiting...
> >>>>>>>>>>
> >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the file
> >>>>>>>>>> in this case reads:
> >>>>>>>>>>
> >>>>>>>>>> 14 (t14a2)
> >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> wgrib2 oz1hr_DAY1
> >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> >>>>>>>>>> 15 (t14a2)
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>>>>>>>>
> >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
suggested
> >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe it
> needs to be
> >>>>>>>>>> the time stamp of the first day?
> >>>>>>>>>>
> >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for any thoughts.
> >>>>>>>>>>
> >>>>>>>>>> Perry
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> >>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Perry,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm confused. In the previous air quality vx we did, we
used
> >>>>>>>>>>> observation files named:
> >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> >>>>>>>>>>>
> >>>>>>>>>>> And those files don't exist in the July data you just
sent.
> >>>>>>>>>>> Instead, it's BUFR files with names like:
> >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure if you're actually verifying against the
BUFR or
> >>>>>>>>>>> PREPBUFR.
> >>>>>>>>>>>
> >>>>>>>>>>> Let me just tell you the key points for the config file
> >>>>>>>>>>> settings...
> >>>>>>>>>>>
> >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files -
> >>>>>>>>>>> thecurrent day and the previous day...
> >>>>>>>>>>> pb2nc \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> >>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>
> >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
like
> this:
> >>>>>>>>>>> time_summary = {
> >>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>> beg = "040000";
> >>>>>>>>>>> end = "040000";
> >>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> >>>>>>>>>>> 23-hours (05Z to 04Z)
> >>>>>>>>>>> grib_code = [];
> >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> >>>>>>>>>>> type = [ "max", "mean" ];
> >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at least
> >>>>>>>>>>> 75% of expect obs are present
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>> That'll compute the MAX and MEAN values at each station
over
> the
> >>>>>>>>>>> defined time period. They're identified in the output
by the
> message type:
> >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> >>>>>>>>>>>
> >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead of
> specifying
> >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient to
> specify the
> >>>>>>>>>>> valid time. So that you don't have to modify the config
file,
> I'd suggest
> >>>>>>>>>>> using environment variables:
> >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> >>>>>>>>>>>
> >>>>>>>>>>> fcst = {
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>> obs = {
> >>>>>>>>>>> convert(x) = x * 10^9;
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> >>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Sounds good, thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> John
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> Affiliate <
> >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids and
focus
> on
> >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
over to
> Theia so we'll
> >>>>>>>>>>>>> let him work on the files he has for now and if he
runs into
> problems, I'll
> >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
max/avg
> files).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not for
bias
> >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
you may
> want to provide
> >>>>>>>>>>>>>> him para5 runs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John to do
> >>>>>>>>>>>>>> some testing some production files. Do you think I
should
> use parallels
> >>>>>>>>>>>>>> instead or do you think the ops files are OK for
creating a
> MET+ use case
> >>>>>>>>>>>>>> for AQ?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> should
> >>>>>>>>>>>>>> be OK for now.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
We
> >>>>>>>>>>>>>>> wanted to update the error in daily max hour labels
but
> management did not
> >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> >>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA Affiliate
wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John over
> parallel CMAQ data
> >>>>>>>>>>>>>>> rather than production CMAQ.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions of
> >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they can
also
> clarify this as
> >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does over
the
> current wgrib2?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to use
the
> >>>>>>>>>>>>>>>> latest parallel run output (para5).
> >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall Jianping
also
> >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
did not
> want to risk any
> >>>>>>>>>>>>>>>> further delays so we will have to do so with the
FYQ4
> implementation.
> >>>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA Affiliate
wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think I
> >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago, but
> I don't know if I
> >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I think
the
> parallel CMAQ they
> >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> But he would have
> >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> this.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
talks
> >>>>>>>>>>>>>>>> about?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley Gotway
<
> >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Perry,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute the
> >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
hour
> ozone concentrations.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> pb2nc_output.nc
> >>>>>>>>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
\
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
\
> >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Using the attached config files.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
FULL,
> for interpolation
> >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But I noticed some oddities:
> >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
this
> "max"
> >>>>>>>>>>>>>>>>> file:
> >>>>>>>>>>>>>>>>> wgrib2
> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-40
hour
> >>>>>>>>>>>>>>>>> ave fcst:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
revealed
> >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17 and
> 41, respectively.
> >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the same
file,
> we
> >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which one
to
> use. Do this by
> >>>>>>>>>>>>>>>>> defining the "lead_time":
> >>>>>>>>>>>>>>>>> fcst = {
> >>>>>>>>>>>>>>>>> field = [
> >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> >>>>>>>>>>>>>>>>> "41";' }
> >>>>>>>>>>>>>>>>> ];
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off by 1
> >>>>>>>>>>>>>>>>> hour.
> >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
time =
> >>>>>>>>>>>>>>>>> valid time of 2018052805.
> >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max value
from
> >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z while
the
> obs are valid at 04Z.
> >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat config
file
> by
> >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> >>>>>>>>>>>>>>>>> obs_window = {
> >>>>>>>>>>>>>>>>> beg = -3600;
> >>>>>>>>>>>>>>>>> end = -3600;
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So what're the next steps?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We're making progress on the time summaries, but
I found
> >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute 24-
hour
> >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output to
> an ascii file using
> >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to count
up the
> number of regular
> >>>>>>>>>>>>>>>>>> and maximum obs values:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
wc -l
> >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
sort
> -u
> >>>>>>>>>>>>>>>>>> | wc -l
> >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt | wc
-l
> >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
sort -u |
> >>>>>>>>>>>>>>>>>> wc -l
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
all of
> >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
meaning
> >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
duplicates.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> For example, the last station ID in that file is
named
> >>>>>>>>>>>>>>>>>> "2FB31EEC":
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
2FB31EEC
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
-114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in the
last
> two
> >>>>>>>>>>>>>>>>>> lines.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates are
> >>>>>>>>>>>>>>>>>> showing up?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers.
> >>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
we'd have
> >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across 2
> files. Do we have existing
> >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
compute
> a 24-hour max
> >>>>>>>>>>>>>>>>>>> temperature or something?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319 sec).
> >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in the
new
> >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content of
> obs_hid
> >>>>>>>>>>>>>>>>>>>>> variable).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
(test
> >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
added
> one
> >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> check the unit test today.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change the
nightly
> >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> covered? If not, should I add
> >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
data.
> >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The output is available
> >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
Gotway <
> >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic for
PB2NC.
> >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> maximum values of COPO from
> >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr counts:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to dump
to
> ascii
> >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000 string
show
> up in the output.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC not
> actually
> >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
Rscript
> dumping the data.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John Halley
Gotway
> <
> >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard Soh
<
> >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
57246
> observations
> >>>>>>>>>>>>>>>>>>>>>>>>> from aqm.20180528/aqm.t12z.prepbufr.tm00.
There
> >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
I'm
> working on this.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> Gotway
> >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
take a
> >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC. I
> copied the data over to
> >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
maximums
> >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end = 0;
} //
> >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what you'd
> expect. The hdr_typ_table
> >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the "AIRNOW_MAX_240000"
> message type, but as you pointed
> >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused by
using 2
> >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> cover the desired 04Z to 04Z
> >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW PrepBUFR
file
> >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
hour
> maximum of obs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is reasonable
for me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data in
NetCDF
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> "AIRNOW_MAX_240000".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
generated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
than obs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
output:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for summary)
exists
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable. The
first
> header index for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1 -d" "
| sort
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota in
here:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180622
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all of
the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
them
> should actually be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table pb2nc_output.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
lookup
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Apr 26 10:25:44 2019
Hi, David,
I put all you need in the directory /incoming/irap/met_help/pshafran
on
your ftp server.
Let me know if you require more items or need some explanation of what
I
have here.
Thanks!
Perry
On Fri, Apr 26, 2019 at 12:08 PM David Fillmore via RT
<met_help at ucar.edu>
wrote:
> Hi Perry -
>
> I'm also forwarding this ticket to Howard,
> I think John is quite busy with the MET 8.1 release so it might be
later
> next week before he gets a chance to look at this.
>
> If you want to upload your data and config files to our ftp server
we can
> replicate and debug the error here.
> I can have a first look this afternoon.
> You can create a subdir at
> ftp.rap.ucar.edu://incoming/irap/met_help
>
> thanks,
> David
>
> On Thu, Apr 25, 2019 at 11:17 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Apr 25 11:16:42 2019: Request 89921 was acted upon.
> > Transaction: Ticket created by perry.shafran at noaa.gov
> > Queue: met_help
> > Subject: Re: PB2NC time summary questions
> > Owner: Nobody
> > Requestors: perry.shafran at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> >
> > Hi, everyone,
> >
> > I am attaching this to this long thread regarding the daily max of
ozone
> > verification, so we remember what we were doing back then.
> >
> > I am now running this again using MET 8.0, and something seems to
not be
> > working when I run point_stat. I get this error when running a
daily max
> > of ozone:
> >
> > ERROR :
> > ERROR : StringArray::operator[](int) const -> range check error!
> > ERROR :
> >
> > I am using my own scripts to run this using MET, so this is not
using
> > METplus.
> >
> > You can find a log file of this here:
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> It
> > runs pb2nc and point_stat for max ozone. The pb2nc code is working
fine,
> I
> > believe.
> >
> > The script I am using to run this is here:
> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> >
> > The prepbufr files are:
> >
> > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> >
> > The model file is here:
> >
> >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> >
> >
> > What could be causing this error?
> >
> > Thanks!
> >
> > Perry
> >
> >
> > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Great, glad that did the trick.
> > >
> > > John
> > >
> > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > > perry.shafran at noaa.gov> wrote:
> > >
> > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
avg.
> > >>
> > >> Thanks!
> > >>
> > >> Perry
> > >>
> > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
<johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >>> Perry,
> > >>>
> > >>> I think the problem is in your setting of
> valid_time="${CUR_VX_TIME}";
> > >>>
> > >>> Here's your wgrib2 output:
> > >>> wgrib2 oz_DAY1
> > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > >>>
> > >>> The "d=2018081112" tells you the model initialization time.
The
> valid
> > >>> time = init time + forecast time. So the OZMAX1 data is valid
16
> hours
> > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> wgrib2
> > >>> command using the "-vt" argument to list the verification
time.
> > >>>
> > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> valid
> > >>> at the time you're requesting.
> > >>>
> > >>> John
> > >>>
> > >>>
> > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate
<
> > >>> perry.shafran at noaa.gov> wrote:
> > >>>
> > >>>> Hi, John,
> > >>>>
> > >>>> I haven't been able to work on this for awhile, but now I'm
working
> on
> > >>>> it again but seem to be running into issues.
> > >>>>
> > >>>> I have a file that has both the daily max for 1-hr ozone and
the
> daily
> > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > >>>>
> > >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > >>>> wgrib2 oz_DAY1
> > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > >>>> 10 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > >>>>
> > >>>> As you can see they have different forecast times. But when
I run
> > >>>> point_stat, I get this output:
> > >>>>
> > >>>> DEBUG 2:
> > >>>> DEBUG 2:
> > >>>>
> >
>
--------------------------------------------------------------------------------
> > >>>> DEBUG 2:
> > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > >>>> WARNING:
> > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
records
> > >>>> found for 'OZMAX8/L1'
> > >>>> WARNING:
> > >>>> WARNING:
> > >>>> WARNING: process_fcst_climo_files() -> no fields matching
OZMAX8/L1
> > >>>> found in file: oz_DAY1
> > >>>> WARNING:
> > >>>>
> > >>>> Here is the relevant portion of the point_stat config file:
> > >>>>
> > >>>> fcst = {
> > >>>> field = [
> > >>>> { name = "OZMAX1"; level = "L1";
valid_time="${CUR_VX_TIME}";
> },
> > >>>> { name = "OZMAX8"; level = "L1";
valid_time="${CUR_VX_TIME}";
> }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> obs = {
> > >>>> convert(x) = x * 10^9;
> > >>>> field = [
> > >>>> { name = "COPO"; level = "A1"; message_type =
> > >>>> "AIRNOW_MAX_230000"; },
> > >>>> { name = "COPO"; level = "A8"; message_type =
> > >>>> "AIRNOW_MAX_230000"; }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> I don't remember if I had this working previously for both
the 1hr
> and
> > >>>> the 8hr ozone, but apparently there is some sort of issue for
the
> 8hr
> > >>>> ozone. Have any thoughts?
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>> Perry
> > >>>>
> > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate <
> > >>>> perry.shafran at noaa.gov> wrote:
> > >>>>
> > >>>>> I chose option 2, and that seems to work - I have a .stat
file with
> > >>>>> output. Now I'll need to do some work here to see if the
output
> > looks OK,
> > >>>>> and to configure the verification so it verifies OZMAX1 and
OZMAX8.
> > >>>>>
> > >>>>> I also wonder if this would be easily modified for PM.
> > >>>>>
> > >>>>> Perry
> > >>>>>
> > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway <
> johnhg at ucar.edu
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Perry,
> > >>>>>>
> > >>>>>> Yep, you've configured/run PB2NC to compute time summaries
that
> are
> > >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> > purportedly
> > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > rejected due
> > >>>>>> to the valid time.
> > >>>>>>
> > >>>>>> It's up to you guys to decide what the "correct" way of
handling
> > this
> > >>>>>> is... but here are a couple of options:
> > >>>>>>
> > >>>>>> (1) You could use the existing setup and address this logic
in
> > >>>>>> Point-Stat config file. Change the matching time
obs_window from
> 0
> > to:
> > >>>>>>
> > >>>>>> obs_window = {
> > >>>>>> beg = 3600;
> > >>>>>> end = 3600;
> > >>>>>> }
> > >>>>>> This tells Point-Stat to match observations which are
exactly
> 1-hour
> > >>>>>> prior to the valid time of the forecast.
> > >>>>>>
> > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z to
> 05Z
> > >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> > >>>>>>
> > >>>>>> time_summary = {
> > >>>>>> flag = TRUE;
> > >>>>>> beg = "050000";
> > >>>>>> end = "050000";
> > >>>>>> ...
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> John
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate <
> > >>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>
> > >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
it's
> > >>>>>>> empty. Here's the output from point_stat:
> > >>>>>>>
> > >>>>>>> DEBUG 4:
> > >>>>>>> DEBUG 4: Data plane information:
> > >>>>>>> DEBUG 4: plane min: 13.86
> > >>>>>>> DEBUG 4: plane max: 130.82
> > >>>>>>> DEBUG 4: scan mode: 64
> > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > >>>>>>> DEBUG 4: lead time: 170000
> > >>>>>>> DEBUG 4: init time: 20180708_120000
> > >>>>>>> DEBUG 4: bitmap flag: 255
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> type
> > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > NEAREST(1),
> > >>>>>>> using 0 pairs.
> > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > >>>>>>> DEBUG 3: Observations processed = 124637
> > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > >>>>>>>
> > >>>>>>> Looks like most rejections were due to valid time.
> > >>>>>>>
> > >>>>>>> Have any thoughts about this one?
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>>
> > >>>>>>> Perry
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
Affiliate <
> > >>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>
> > >>>>>>>> Hi, John,
> > >>>>>>>>
> > >>>>>>>> Looks like the changing of the time stamp from _04 to _05
did
> the
> > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > >>>>>>>>
> > >>>>>>>> Thanks!
> > >>>>>>>>
> > >>>>>>>> Perry
> > >>>>>>>>
> > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > >>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>
> > >>>>>>>>> Perry,
> > >>>>>>>>>
> > >>>>>>>>> I agree, there's likely some issue with requesting the
> timestamp
> > >>>>>>>>> of the data from that file. I think switching the
verification
> > time to
> > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > >>>>>>>>>
> > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> times
> > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2 and
> > MET commands
> > >>>>>>>>> to illustrate.
> > >>>>>>>>>
> > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
time
> > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample data
> > file you
> > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2 so I
> > have that
> > >>>>>>>>> funky forecast hour:
> > >>>>>>>>>
> > >>>>>>>>> wgrib2 -t -ftime -vt
> > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > >>>>>>>>> fcst:vt=2018070505
> > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > >>>>>>>>>
> > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level
> (-v
> > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > reading:
> > >>>>>>>>>
> > >>>>>>>>> # Read the first record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > >>>>>>>>> DEBUG 4: lead time: 230000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>> ...
> > >>>>>>>>> # Read the second record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > >>>>>>>>> DEBUG 4: lead time: 470000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>>
> > >>>>>>>>> Hope that helps.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> John
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
Affiliate
> <
> > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, John,
> > >>>>>>>>>>
> > >>>>>>>>>> When I implemented all the changes (and also set it up
so it's
> > >>>>>>>>>> reading the beta6 version), I used this command:
> > >>>>>>>>>>
> > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > >>>>>>>>>>
> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
-v
> 4
> > >>>>>>>>>>
> > >>>>>>>>>> But then I got this error:
> > >>>>>>>>>>
> > >>>>>>>>>> DEBUG 2:
> > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > >>>>>>>>>> records found for 'OZMAX1/L1'
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> ERROR :
> > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> > data
> > >>>>>>>>>> found! Exiting...
> > >>>>>>>>>>
> > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the
> file
> > >>>>>>>>>> in this case reads:
> > >>>>>>>>>>
> > >>>>>>>>>> 14 (t14a2)
> > >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>
> > wgrib2 oz1hr_DAY1
> > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > >>>>>>>>>> 15 (t14a2)
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>>>>>>>>
> > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
suggested
> > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe it
> > needs to be
> > >>>>>>>>>> the time stamp of the first day?
> > >>>>>>>>>>
> > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for any thoughts.
> > >>>>>>>>>>
> > >>>>>>>>>> Perry
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Perry,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm confused. In the previous air quality vx we did,
we used
> > >>>>>>>>>>> observation files named:
> > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > >>>>>>>>>>>
> > >>>>>>>>>>> And those files don't exist in the July data you just
sent.
> > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm not sure if you're actually verifying against the
BUFR or
> > >>>>>>>>>>> PREPBUFR.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > >>>>>>>>>>> settings...
> > >>>>>>>>>>>
> > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files -
> > >>>>>>>>>>> thecurrent day and the previous day...
> > >>>>>>>>>>> pb2nc \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > >>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>
> > >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
like
> > this:
> > >>>>>>>>>>> time_summary = {
> > >>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>> end = "040000";
> > >>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > >>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > >>>>>>>>>>> type = [ "max", "mean" ];
> > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at
> least
> > >>>>>>>>>>> 75% of expect obs are present
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station over
> > the
> > >>>>>>>>>>> defined time period. They're identified in the output
by the
> > message type:
> > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > >>>>>>>>>>>
> > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead
> of
> > specifying
> > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient to
> > specify the
> > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> file,
> > I'd suggest
> > >>>>>>>>>>> using environment variables:
> > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > >>>>>>>>>>>
> > >>>>>>>>>>> fcst = {
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>> obs = {
> > >>>>>>>>>>> convert(x) = x * 10^9;
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Sounds good, thanks!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> John
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> > Affiliate <
> > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and focus
> > on
> > >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
over to
> > Theia so we'll
> > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> into
> > problems, I'll
> > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
max/avg
> > files).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for bias
> > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
you
> may
> > want to provide
> > >>>>>>>>>>>>>> him para5 runs.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John to
> do
> > >>>>>>>>>>>>>> some testing some production files. Do you think I
should
> > use parallels
> > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
creating
> a
> > MET+ use case
> > >>>>>>>>>>>>>> for AQ?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > should
> > >>>>>>>>>>>>>> be OK for now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
We
> > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels but
> > management did not
> > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> > >>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > parallel CMAQ data
> > >>>>>>>>>>>>>>> rather than production CMAQ.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions of
> > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can also
> > clarify this as
> > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over the
> > current wgrib2?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use the
> > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping also
> > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
did
> not
> > want to risk any
> > >>>>>>>>>>>>>>>> further delays so we will have to do so with the
FYQ4
> > implementation.
> > >>>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think I
> > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago, but
> > I don't know if I
> > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think the
> > parallel CMAQ they
> > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> > But he would have
> > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> > this.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
talks
> > >>>>>>>>>>>>>>>> about?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Perry,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute
> the
> > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
hour
> > ozone concentrations.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > pb2nc_output.nc
> > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Using the attached config files.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
FULL,
> > for interpolation
> > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
this
> > "max"
> > >>>>>>>>>>>>>>>>> file:
> > >>>>>>>>>>>>>>>>> wgrib2
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-
40 hour
> > >>>>>>>>>>>>>>>>> ave fcst:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
> revealed
> > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17 and
> > 41, respectively.
> > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same file,
> > we
> > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one to
> > use. Do this by
> > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > >>>>>>>>>>>>>>>>> fcst = {
> > >>>>>>>>>>>>>>>>> field = [
> > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> > >>>>>>>>>>>>>>>>> "41";' }
> > >>>>>>>>>>>>>>>>> ];
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off by
> 1
> > >>>>>>>>>>>>>>>>> hour.
> > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
time =
> > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value from
> > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while the
> > obs are valid at 04Z.
> > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config file
> > by
> > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > >>>>>>>>>>>>>>>>> obs_window = {
> > >>>>>>>>>>>>>>>>> beg = -3600;
> > >>>>>>>>>>>>>>>>> end = -3600;
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> So what're the next steps?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> found
> > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> > >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-hour
> > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output to
> > an ascii file using
> > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count up
> the
> > number of regular
> > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
| wc
> -l
> > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
| sort
> > -u
> > >>>>>>>>>>>>>>>>>> | wc -l
> > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
wc -l
> > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
sort
> -u |
> > >>>>>>>>>>>>>>>>>> wc -l
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
all of
> > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
> meaning
> > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> duplicates.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is named
> > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
> 2FB31EEC
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the last
> > two
> > >>>>>>>>>>>>>>>>>> lines.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates are
> > >>>>>>>>>>>>>>>>>> showing up?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Cheers.
> > >>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
we'd
> have
> > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across
2
> > files. Do we have existing
> > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
compute
> > a 24-hour max
> > >>>>>>>>>>>>>>>>>>> temperature or something?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> sec).
> > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the new
> > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
<hsoh at ucar.edu
> >
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content
of
> > obs_hid
> > >>>>>>>>>>>>>>>>>>>>> variable).
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
(test
> > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
added
> > one
> > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > check the unit test today.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> nightly
> > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> > covered? If not, should I add
> > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
data.
> > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
> Gotway <
> > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> PB2NC.
> > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> > maximum values of COPO from
> > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump to
> > ascii
> > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string show
> > up in the output.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > actually
> > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
Rscript
> > dumping the data.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> Gotway
> > <
> > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
57246
> > observations
> > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00. There
> > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
I'm
> > working on this.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > Gotway
> > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
take a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC.
I
> > copied the data over to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> \
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
> maximums
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end =
0; } //
> > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > expect. The hdr_typ_table
> > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > message type, but as you pointed
> > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> using 2
> > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> > cover the desired 04Z to 04Z
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR file
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
hour
> > maximum of obs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable for
> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> NetCDF
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > "AIRNOW_MAX_240000".
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
generated.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
than
> obs
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
output:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> currently)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> exists
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The first
> > header index for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1 -d"
" |
> sort
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota
in
> here:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180622
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > svn-met-dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
them
> > should actually be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
lookup
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Fri Apr 26 10:41:06 2019
Thanks Perry,
We have your files and will have a look...
David
On Fri, Apr 26, 2019 at 10:26 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, David,
>
> I put all you need in the directory /incoming/irap/met_help/pshafran
on
> your ftp server.
>
> Let me know if you require more items or need some explanation of
what I
> have here.
>
> Thanks!
>
> Perry
>
> On Fri, Apr 26, 2019 at 12:08 PM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Perry -
> >
> > I'm also forwarding this ticket to Howard,
> > I think John is quite busy with the MET 8.1 release so it might be
later
> > next week before he gets a chance to look at this.
> >
> > If you want to upload your data and config files to our ftp server
we can
> > replicate and debug the error here.
> > I can have a first look this afternoon.
> > You can create a subdir at
> > ftp.rap.ucar.edu://incoming/irap/met_help
> >
> > thanks,
> > David
> >
> > On Thu, Apr 25, 2019 at 11:17 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Apr 25 11:16:42 2019: Request 89921 was acted upon.
> > > Transaction: Ticket created by perry.shafran at noaa.gov
> > > Queue: met_help
> > > Subject: Re: PB2NC time summary questions
> > > Owner: Nobody
> > > Requestors: perry.shafran at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
> >
> > >
> > >
> > > Hi, everyone,
> > >
> > > I am attaching this to this long thread regarding the daily max
of
> ozone
> > > verification, so we remember what we were doing back then.
> > >
> > > I am now running this again using MET 8.0, and something seems
to not
> be
> > > working when I run point_stat. I get this error when running a
daily
> max
> > > of ozone:
> > >
> > > ERROR :
> > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > ERROR :
> > >
> > > I am using my own scripts to run this using MET, so this is not
using
> > > METplus.
> > >
> > > You can find a log file of this here:
> > >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > It
> > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> fine,
> > I
> > > believe.
> > >
> > > The script I am using to run this is here:
> > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > >
> > > The prepbufr files are:
> > >
> > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > >
> > > The model file is here:
> > >
> > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > >
> > >
> > > What could be causing this error?
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > >
> > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Great, glad that did the trick.
> > > >
> > > > John
> > > >
> > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate
<
> > > > perry.shafran at noaa.gov> wrote:
> > > >
> > > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-
hr avg.
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Perry
> > > >>
> > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway <
> johnhg at ucar.edu>
> > > >> wrote:
> > > >>
> > > >>> Perry,
> > > >>>
> > > >>> I think the problem is in your setting of
> > valid_time="${CUR_VX_TIME}";
> > > >>>
> > > >>> Here's your wgrib2 output:
> > > >>> wgrib2 oz_DAY1
> > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > >>>
> > > >>> The "d=2018081112" tells you the model initialization time.
The
> > valid
> > > >>> time = init time + forecast time. So the OZMAX1 data is
valid 16
> > hours
> > > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> > wgrib2
> > > >>> command using the "-vt" argument to list the verification
time.
> > > >>>
> > > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > valid
> > > >>> at the time you're requesting.
> > > >>>
> > > >>> John
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate <
> > > >>> perry.shafran at noaa.gov> wrote:
> > > >>>
> > > >>>> Hi, John,
> > > >>>>
> > > >>>> I haven't been able to work on this for awhile, but now I'm
> working
> > on
> > > >>>> it again but seem to be running into issues.
> > > >>>>
> > > >>>> I have a file that has both the daily max for 1-hr ozone
and the
> > daily
> > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > >>>>
> > > >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > > >>>> wgrib2 oz_DAY1
> > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > >>>> 10 (t10a2)
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > >>>>
> > > >>>> As you can see they have different forecast times. But
when I run
> > > >>>> point_stat, I get this output:
> > > >>>>
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2:
> > > >>>>
> > >
> >
>
--------------------------------------------------------------------------------
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > >>>> WARNING:
> > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> records
> > > >>>> found for 'OZMAX8/L1'
> > > >>>> WARNING:
> > > >>>> WARNING:
> > > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> OZMAX8/L1
> > > >>>> found in file: oz_DAY1
> > > >>>> WARNING:
> > > >>>>
> > > >>>> Here is the relevant portion of the point_stat config file:
> > > >>>>
> > > >>>> fcst = {
> > > >>>> field = [
> > > >>>> { name = "OZMAX1"; level = "L1";
> valid_time="${CUR_VX_TIME}";
> > },
> > > >>>> { name = "OZMAX8"; level = "L1";
> valid_time="${CUR_VX_TIME}";
> > }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> obs = {
> > > >>>> convert(x) = x * 10^9;
> > > >>>> field = [
> > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; },
> > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> I don't remember if I had this working previously for both
the 1hr
> > and
> > > >>>> the 8hr ozone, but apparently there is some sort of issue
for the
> > 8hr
> > > >>>> ozone. Have any thoughts?
> > > >>>>
> > > >>>> Thanks!
> > > >>>>
> > > >>>> Perry
> > > >>>>
> > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate <
> > > >>>> perry.shafran at noaa.gov> wrote:
> > > >>>>
> > > >>>>> I chose option 2, and that seems to work - I have a .stat
file
> with
> > > >>>>> output. Now I'll need to do some work here to see if the
output
> > > looks OK,
> > > >>>>> and to configure the verification so it verifies OZMAX1
and
> OZMAX8.
> > > >>>>>
> > > >>>>> I also wonder if this would be easily modified for PM.
> > > >>>>>
> > > >>>>> Perry
> > > >>>>>
> > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Perry,
> > > >>>>>>
> > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries that
> > are
> > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that are
> > > purportedly
> > > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > > rejected due
> > > >>>>>> to the valid time.
> > > >>>>>>
> > > >>>>>> It's up to you guys to decide what the "correct" way of
handling
> > > this
> > > >>>>>> is... but here are a couple of options:
> > > >>>>>>
> > > >>>>>> (1) You could use the existing setup and address this
logic in
> > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> from
> > 0
> > > to:
> > > >>>>>>
> > > >>>>>> obs_window = {
> > > >>>>>> beg = 3600;
> > > >>>>>> end = 3600;
> > > >>>>>> }
> > > >>>>>> This tells Point-Stat to match observations which are
exactly
> > 1-hour
> > > >>>>>> prior to the valid time of the forecast.
> > > >>>>>>
> > > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z to
> > 05Z
> > > >>>>>> instead and then leave the Point-Stat obs_window setting
at 0:
> > > >>>>>>
> > > >>>>>> time_summary = {
> > > >>>>>> flag = TRUE;
> > > >>>>>> beg = "050000";
> > > >>>>>> end = "050000";
> > > >>>>>> ...
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> John
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate
> <
> > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>
> > > >>>>>>> Oops, spoke too soon. There *is* a .stat output file,
but it's
> > > >>>>>>> empty. Here's the output from point_stat:
> > > >>>>>>>
> > > >>>>>>> DEBUG 4:
> > > >>>>>>> DEBUG 4: Data plane information:
> > > >>>>>>> DEBUG 4: plane min: 13.86
> > > >>>>>>> DEBUG 4: plane max: 130.82
> > > >>>>>>> DEBUG 4: scan mode: 64
> > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > >>>>>>> DEBUG 4: lead time: 170000
> > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > type
> > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > > NEAREST(1),
> > > >>>>>>> using 0 pairs.
> > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > >>>>>>>
> > > >>>>>>> Looks like most rejections were due to valid time.
> > > >>>>>>>
> > > >>>>>>> Have any thoughts about this one?
> > > >>>>>>>
> > > >>>>>>> Thanks!
> > > >>>>>>>
> > > >>>>>>> Perry
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> Affiliate <
> > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi, John,
> > > >>>>>>>>
> > > >>>>>>>> Looks like the changing of the time stamp from _04 to
_05 did
> > the
> > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > >>>>>>>>
> > > >>>>>>>> Thanks!
> > > >>>>>>>>
> > > >>>>>>>> Perry
> > > >>>>>>>>
> > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Perry,
> > > >>>>>>>>>
> > > >>>>>>>>> I agree, there's likely some issue with requesting the
> > timestamp
> > > >>>>>>>>> of the data from that file. I think switching the
> verification
> > > time to
> > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > >>>>>>>>>
> > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > times
> > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2 and
> > > MET commands
> > > >>>>>>>>> to illustrate.
> > > >>>>>>>>>
> > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast time
> > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> data
> > > file you
> > > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> so I
> > > have that
> > > >>>>>>>>> funky forecast hour:
> > > >>>>>>>>>
> > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > >>>>>>>>> fcst:vt=2018070505
> > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > >>>>>>>>>
> > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level
> > (-v
> > > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > > reading:
> > > >>>>>>>>>
> > > >>>>>>>>> # Read the first record
> > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>> ...
> > > >>>>>>>>> # Read the second record
> > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>>
> > > >>>>>>>>> Hope that helps.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> John
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> Affiliate
> > <
> > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi, John,
> > > >>>>>>>>>>
> > > >>>>>>>>>> When I implemented all the changes (and also set it
up so
> it's
> > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > >>>>>>>>>>
> > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > >>>>>>>>>>
> > >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> -v
> > 4
> > > >>>>>>>>>>
> > > >>>>>>>>>> But then I got this error:
> > > >>>>>>>>>>
> > > >>>>>>>>>> DEBUG 2:
> > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> ERROR :
> > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> > > data
> > > >>>>>>>>>> found! Exiting...
> > > >>>>>>>>>>
> > > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output
for the
> > file
> > > >>>>>>>>>> in this case reads:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 14 (t14a2)
> > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > wgrib2 oz1hr_DAY1
> > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>>>>>>>>> 15 (t14a2)
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>>>>>>>>
> > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> suggested
> > > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> it
> > > needs to be
> > > >>>>>>>>>> the time stamp of the first day?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for any thoughts.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Perry
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Perry,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did, we
> used
> > > >>>>>>>>>>> observation files named:
> > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> And those files don't exist in the July data you
just sent.
> > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm not sure if you're actually verifying against
the BUFR
> or
> > > >>>>>>>>>>> PREPBUFR.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > > >>>>>>>>>>> settings...
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files
> -
> > > >>>>>>>>>>> thecurrent day and the previous day...
> > > >>>>>>>>>>> pb2nc \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > >>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary like
> > > this:
> > > >>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > >>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries
when at
> > least
> > > >>>>>>>>>>> 75% of expect obs are present
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> over
> > > the
> > > >>>>>>>>>>> defined time period. They're identified in the
output by
> the
> > > message type:
> > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > > >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead
> > of
> > > specifying
> > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> to
> > > specify the
> > > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > file,
> > > I'd suggest
> > > >>>>>>>>>>> using environment variables:
> > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> fcst = {
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>> obs = {
> > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Sounds good, thanks!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> John
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > Affiliate <
> > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and
> focus
> > > on
> > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files over
> to
> > > Theia so we'll
> > > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> > into
> > > problems, I'll
> > > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-
hr
> max/avg
> > > files).
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for
> bias
> > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe, you
> > may
> > > want to provide
> > > >>>>>>>>>>>>>> him para5 runs.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> to
> > do
> > > >>>>>>>>>>>>>> some testing some production files. Do you think
I
> should
> > > use parallels
> > > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> creating
> > a
> > > MET+ use case
> > > >>>>>>>>>>>>>> for AQ?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > > should
> > > >>>>>>>>>>>>>> be OK for now.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels. We
> > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels but
> > > management did not
> > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > >>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > > parallel CMAQ data
> > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions
> of
> > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can
> also
> > > clarify this as
> > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over the
> > > current wgrib2?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use
> the
> > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> also
> > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management did
> > not
> > > want to risk any
> > > >>>>>>>>>>>>>>>> further delays so we will have to do so with
the FYQ4
> > > implementation.
> > > >>>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think
> I
> > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> but
> > > I don't know if I
> > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> the
> > > parallel CMAQ they
> > > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> > > But he would have
> > > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> > > this.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues
John
> talks
> > > >>>>>>>>>>>>>>>> about?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute
> > the
> > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and
8-hour
> > > ozone concentrations.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region FULL,
> > > for interpolation
> > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents
of this
> > > "max"
> > > >>>>>>>>>>>>>>>>> file:
> > > >>>>>>>>>>>>>>>>> wgrib2
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
level:17-40
> hour
> > > >>>>>>>>>>>>>>>>> ave fcst:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > revealed
> > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours 17
> and
> > > 41, respectively.
> > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> file,
> > > we
> > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one to
> > > use. Do this by
> > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > >>>>>>>>>>>>>>>>> fcst = {
> > > >>>>>>>>>>>>>>>>> field = [
> > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> > > >>>>>>>>>>>>>>>>> "41";' }
> > > >>>>>>>>>>>>>>>>> ];
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps
are off
> by
> > 1
> > > >>>>>>>>>>>>>>>>> hour.
> > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead time
> =
> > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> from
> > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> the
> > > obs are valid at 04Z.
> > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> file
> > > by
> > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > > >>>>>>>>>>>>>>>>> obs_window = {
> > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > >>>>>>>>>>>>>>>>> end = -3600;
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> > found
> > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-hour
> > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> to
> > > an ascii file using
> > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count up
> > the
> > > number of regular
> > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt | wc
> > -l
> > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> sort
> > > -u
> > > >>>>>>>>>>>>>>>>>> | wc -l
> > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt
| wc -l
> > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt
| sort
> > -u |
> > > >>>>>>>>>>>>>>>>>> wc -l
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning
that all
> of
> > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > meaning
> > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > duplicates.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is
> named
> > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > 2FB31EEC
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the
> last
> > > two
> > > >>>>>>>>>>>>>>>>>> lines.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> are
> > > >>>>>>>>>>>>>>>>>> showing up?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu
> >
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > >>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason
why we'd
> > have
> > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > files. Do we have existing
> > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> compute
> > > a 24-hour max
> > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > sec).
> > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the
> new
> > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh <
> hsoh at ucar.edu
> > >
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content of
> > > obs_hid
> > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> (test
> > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs.
I
> added
> > > one
> > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > > check the unit test today.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> > nightly
> > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > covered? If not, should I add
> > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh
<
> > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> data.
> > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > Gotway <
> > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> > PB2NC.
> > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute
24-hour
> > > maximum values of COPO from
> > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump to
> > > ascii
> > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> show
> > > up in the output.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > > actually
> > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in
the
> Rscript
> > > dumping the data.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > Gotway
> > > <
> > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00
and
> 57246
> > > observations
> > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> There
> > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data. I'm
> > > working on this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > > Gotway
> > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> take a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC. I
> > > copied the data over to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > maximums
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end
= 0; }
> //
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > > expect. The hdr_typ_table
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > > message type, but as you pointed
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> > using 2
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of
them to
> > > cover the desired 04Z to 04Z
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> file
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a
24 hour
> > > maximum of obs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh
<
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable for
> > me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> > NetCDF
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > "AIRNOW_MAX_240000".
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> generated.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers than
> > obs
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> output:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> > currently)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0
to
> 3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > exists
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The
> first
> > > header index for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt | cut -f1
-d" " |
> > sort
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > > wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota in
> > here:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > svn-met-
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some
of them
> > > should actually be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced
by this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> lookup
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Thu May 02 15:35:10 2019
Hi Perry -
An update on where we are on this issue:
Using the prepda.2019042312.nc file you provided we indeed reproduce
the point_stat error.
The point_stat error is due to only a single entry the valid time
table,
e.g. from ncdump -v hdr_vld_table
hdr_vld_table =
"20190423_050000",
while the hdr_vld indices in that file are hdr_vld = 0, 33, 33, ...
which are all out of range except the first.
If I run pb2nc here with your PREPBUFR data and your PB2NCConfig_aqm
config file, however,
I generate a different prepda.2019042312.nc file that runs through to
completion with point_stat.
Its valid time table:
hdr_vld_table =
"20190423_030000",
"20190423_010000",
"20190423_020000",
"20190423_230000",
"20190423_150000",
"20190423_090000",
"20190423_070000",
"20190423_060000",
"20190423_210000",
"20190423_050000",
"20190423_170000",
"20190423_100000",
"20190423_200000",
"20190423_140000",
"20190423_130000",
"20190423_160000",
"20190423_040000",
"20190423_180000",
"20190423_080000",
"20190423_120000",
"20190423_110000",
"20190423_220000",
"20190423_190000",
"20190424_000000" ;
}
So it seems pb2nc is running correctly here, but not on your side.
Let me discuss with John the best way to proceed ...
thanks,
David
On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> Hi, everyone,
>
> I am attaching this to this long thread regarding the daily max of
> ozone
> verification, so we remember what we were doing back then.
>
> I am now running this again using MET 8.0, and something seems to
not
> be
> working when I run point_stat. I get this error when running a
daily
> max
> of ozone:
>
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> I am using my own scripts to run this using MET, so this is not
using
> METplus.
>
> You can find a log file of this here:
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> It
> runs pb2nc and point_stat for max ozone. The pb2nc code is working
> fine, I
> believe.
>
> The script I am using to run this is here:
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
>
> The prepbufr files are:
>
> /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
>
> The model file is here:
>
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
>
>
> What could be causing this error?
>
> Thanks!
>
> Perry
>
>
> On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Great, glad that did the trick.
> >
> > John
> >
> > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > perry.shafran at noaa.gov> wrote:
> >
> >> Well this worked! And Jeff confirmed to me that the 8-hr average
> >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
avg.
> >>
> >> Thanks!
> >>
> >> Perry
> >>
> >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> >> <johnhg at ucar.edu>
> >> wrote:
> >>
> >>> Perry,
> >>>
> >>> I think the problem is in your setting of
> >>> valid_time="${CUR_VX_TIME}";
> >>>
> >>> Here's your wgrib2 output:
> >>> wgrib2 oz_DAY1
> >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> >>>
> >>> The "d=2018081112" tells you the model initialization time. The
> >>> valid
> >>> time = init time + forecast time. So the OZMAX1 data is valid
16
> >>> hours
> >>> later while the OZMAX8 data is valid 22 hours later. Rerun that
> >>> wgrib2
> >>> command using the "-vt" argument to list the verification time.
> >>>
> >>> Point-Stat isn't finding any OZMAX8 data because that data isn't
> >>> valid
> >>> at the time you're requesting.
> >>>
> >>> John
> >>>
> >>>
> >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate <
> >>> perry.shafran at noaa.gov> wrote:
> >>>
> >>>> Hi, John,
> >>>>
> >>>> I haven't been able to work on this for awhile, but now I'm
> >>>> working on
> >>>> it again but seem to be running into issues.
> >>>>
> >>>> I have a file that has both the daily max for 1-hr ozone and
the
> >>>> daily
> >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> >>>>
> >>>> 9 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>> >
> >>>> wgrib2 oz_DAY1
> >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave fcst:
> >>>> 10 (t10a2)
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>> >
> >>>>
> >>>> As you can see they have different forecast times. But when I
run
> >>>> point_stat, I get this output:
> >>>>
> >>>> DEBUG 2:
> >>>> DEBUG 2:
> >>>>
--------------------------------------------------------------------------------
> >>>> DEBUG 2:
> >>>> DEBUG 2: Reading data for OZMAX8/L1.
> >>>> WARNING:
> >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> >>>> records
> >>>> found for 'OZMAX8/L1'
> >>>> WARNING:
> >>>> WARNING:
> >>>> WARNING: process_fcst_climo_files() -> no fields matching
> >>>> OZMAX8/L1
> >>>> found in file: oz_DAY1
> >>>> WARNING:
> >>>>
> >>>> Here is the relevant portion of the point_stat config file:
> >>>>
> >>>> fcst = {
> >>>> field = [
> >>>> { name = "OZMAX1"; level = "L1";
> >>>> valid_time="${CUR_VX_TIME}"; },
> >>>> { name = "OZMAX8"; level = "L1";
> >>>> valid_time="${CUR_VX_TIME}"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> obs = {
> >>>> convert(x) = x * 10^9;
> >>>> field = [
> >>>> { name = "COPO"; level = "A1"; message_type =
> >>>> "AIRNOW_MAX_230000"; },
> >>>> { name = "COPO"; level = "A8"; message_type =
> >>>> "AIRNOW_MAX_230000"; }
> >>>> ];
> >>>> }
> >>>>
> >>>> I don't remember if I had this working previously for both the
1hr
> >>>> and
> >>>> the 8hr ozone, but apparently there is some sort of issue for
the
> >>>> 8hr
> >>>> ozone. Have any thoughts?
> >>>>
> >>>> Thanks!
> >>>>
> >>>> Perry
> >>>>
> >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA Affiliate
<
> >>>> perry.shafran at noaa.gov> wrote:
> >>>>
> >>>>> I chose option 2, and that seems to work - I have a .stat file
> >>>>> with
> >>>>> output. Now I'll need to do some work here to see if the
output
> >>>>> looks OK,
> >>>>> and to configure the verification so it verifies OZMAX1 and
> >>>>> OZMAX8.
> >>>>>
> >>>>> I also wonder if this would be easily modified for PM.
> >>>>>
> >>>>> Perry
> >>>>>
> >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> >>>>> <johnhg at ucar.edu>
> >>>>> wrote:
> >>>>>
> >>>>>> Perry,
> >>>>>>
> >>>>>> Yep, you've configured/run PB2NC to compute time summaries
that
> >>>>>> are
> >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> >>>>>> purportedly
> >>>>>> valid at 05Z. And that's the reason why you have so many obs
> >>>>>> rejected due
> >>>>>> to the valid time.
> >>>>>>
> >>>>>> It's up to you guys to decide what the "correct" way of
handling
> >>>>>> this
> >>>>>> is... but here are a couple of options:
> >>>>>>
> >>>>>> (1) You could use the existing setup and address this logic
in
> >>>>>> Point-Stat config file. Change the matching time obs_window
> >>>>>> from 0 to:
> >>>>>>
> >>>>>> obs_window = {
> >>>>>> beg = 3600;
> >>>>>> end = 3600;
> >>>>>> }
> >>>>>> This tells Point-Stat to match observations which are exactly
1-
> >>>>>> hour
> >>>>>> prior to the valid time of the forecast.
> >>>>>>
> >>>>>> (2) You could rerun PB2NC to compute a time summary from 05Z
to
> >>>>>> 05Z
> >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> >>>>>>
> >>>>>> time_summary = {
> >>>>>> flag = TRUE;
> >>>>>> beg = "050000";
> >>>>>> end = "050000";
> >>>>>> ...
> >>>>>>
> >>>>>> Thanks,
> >>>>>> John
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
Affiliate
> >>>>>> <
> >>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>
> >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
it's
> >>>>>>> empty. Here's the output from point_stat:
> >>>>>>>
> >>>>>>> DEBUG 4:
> >>>>>>> DEBUG 4: Data plane information:
> >>>>>>> DEBUG 4: plane min: 13.86
> >>>>>>> DEBUG 4: plane max: 130.82
> >>>>>>> DEBUG 4: scan mode: 64
> >>>>>>> DEBUG 4: valid time: 20180709_050000
> >>>>>>> DEBUG 4: lead time: 170000
> >>>>>>> DEBUG 4: init time: 20180708_120000
> >>>>>>> DEBUG 4: bitmap flag: 255
> >>>>>>>
> >>>>>>>
> >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> >>>>>>> type
> >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> >>>>>>> NEAREST(1),
> >>>>>>> using 0 pairs.
> >>>>>>> DEBUG 3: Number of matched pairs = 0
> >>>>>>> DEBUG 3: Observations processed = 124637
> >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> >>>>>>> DEBUG 3: Rejected: obs type = 0
> >>>>>>> DEBUG 3: Rejected: valid time = 122154
> >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> >>>>>>> DEBUG 3: Rejected: off the grid = 70
> >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> >>>>>>> DEBUG 3: Rejected: quality marker = 0
> >>>>>>> DEBUG 3: Rejected: message type = 1189
> >>>>>>> DEBUG 3: Rejected: masking region = 0
> >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> >>>>>>> DEBUG 3: Rejected: duplicates = 0
> >>>>>>>
> >>>>>>> Looks like most rejections were due to valid time.
> >>>>>>>
> >>>>>>> Have any thoughts about this one?
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> Perry
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
Affiliate
> >>>>>>> <
> >>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>
> >>>>>>>> Hi, John,
> >>>>>>>>
> >>>>>>>> Looks like the changing of the time stamp from _04 to _05
did
> >>>>>>>> the
> >>>>>>>> trick as point_stat ran without an issue with that change.
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> Perry
> >>>>>>>>
> >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> >>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>
> >>>>>>>>> Perry,
> >>>>>>>>>
> >>>>>>>>> I agree, there's likely some issue with requesting the
> >>>>>>>>> timestamp
> >>>>>>>>> of the data from that file. I think switching the
> >>>>>>>>> verification time to
> >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> >>>>>>>>>
> >>>>>>>>> Looks like wgrib2 and MET both agree on the initialization
> >>>>>>>>> times
> >>>>>>>>> and valid times for this data. Here's some useful wgrib2
and
> >>>>>>>>> MET commands
> >>>>>>>>> to illustrate.
> >>>>>>>>>
> >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
time
> >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a sample
> >>>>>>>>> data file you
> >>>>>>>>> sent to me. But remember I'm still using the "old" wgrib2
so
> >>>>>>>>> I have that
> >>>>>>>>> funky forecast hour:
> >>>>>>>>>
> >>>>>>>>> wgrib2 -t -ftime -vt
> >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> >>>>>>>>> fcst:vt=2018070505
> >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
> >>>>>>>>>
> >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
level
> >>>>>>>>> (-v
> >>>>>>>>> 4) to have MET tell you the timing info of the data it is
> >>>>>>>>> reading:
> >>>>>>>>>
> >>>>>>>>> # Read the first record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> >>>>>>>>> DEBUG 4: lead time: 230000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>> ...
> >>>>>>>>> # Read the second record
> >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> >>>>>>>>> ...
> >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> >>>>>>>>> DEBUG 4: lead time: 470000
> >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >>>>>>>>>
> >>>>>>>>> Hope that helps.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> John
> >>>>>>>>>
> >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> >>>>>>>>> Affiliate <
> >>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi, John,
> >>>>>>>>>>
> >>>>>>>>>> When I implemented all the changes (and also set it up so
> >>>>>>>>>> it's
> >>>>>>>>>> reading the beta6 version), I used this command:
> >>>>>>>>>>
> >>>>>>>>>> CUR_VX_TIME=20180709_04
> >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> >>>>>>>>>>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> >>>>>>>>>> -v 4
> >>>>>>>>>>
> >>>>>>>>>> But then I got this error:
> >>>>>>>>>>
> >>>>>>>>>> DEBUG 2:
> >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> >>>>>>>>>> records found for 'OZMAX1/L1'
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING:
> >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields matching
> >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> >>>>>>>>>> WARNING:
> >>>>>>>>>> ERROR :
> >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
forecast
> >>>>>>>>>> data
> >>>>>>>>>> found! Exiting...
> >>>>>>>>>>
> >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the
> >>>>>>>>>> file
> >>>>>>>>>> in this case reads:
> >>>>>>>>>>
> >>>>>>>>>> 14 (t14a2)
> >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> >>>>>>>>>> wgrib2 oz1hr_DAY1
> >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> >>>>>>>>>> 15 (t14a2)
> >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >>>>>>>>>>
> >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> >>>>>>>>>> suggested
> >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> >>>>>>>>>> it needs to be
> >>>>>>>>>> the time stamp of the first day?
> >>>>>>>>>>
> >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for any thoughts.
> >>>>>>>>>>
> >>>>>>>>>> Perry
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> >>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Perry,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm confused. In the previous air quality vx we did, we
> >>>>>>>>>>> used
> >>>>>>>>>>> observation files named:
> >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> >>>>>>>>>>>
> >>>>>>>>>>> And those files don't exist in the July data you just
sent.
> >>>>>>>>>>> Instead, it's BUFR files with names like:
> >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.airnow.tm024.bufr_d
> >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure if you're actually verifying against the
BUFR
> >>>>>>>>>>> or
> >>>>>>>>>>> PREPBUFR.
> >>>>>>>>>>>
> >>>>>>>>>>> Let me just tell you the key points for the config file
> >>>>>>>>>>> settings...
> >>>>>>>>>>>
> >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
files
> >>>>>>>>>>> -
> >>>>>>>>>>> thecurrent day and the previous day...
> >>>>>>>>>>> pb2nc \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> >>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>
> >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
like
> >>>>>>>>>>> this:
> >>>>>>>>>>> time_summary = {
> >>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>> beg = "040000";
> >>>>>>>>>>> end = "040000";
> >>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
previous
> >>>>>>>>>>> 23-hours (05Z to 04Z)
> >>>>>>>>>>> grib_code = [];
> >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> >>>>>>>>>>> type = [ "max", "mean" ];
> >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at
> >>>>>>>>>>> least
> >>>>>>>>>>> 75% of expect obs are present
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>> That'll compute the MAX and MEAN values at each station
> >>>>>>>>>>> over the
> >>>>>>>>>>> defined time period. They're identified in the output
by
> >>>>>>>>>>> the message type:
> >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> >>>>>>>>>>>
> >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> >>>>>>>>>>> sections like this for the max of the 1-hour data.
Instead
> >>>>>>>>>>> of specifying
> >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> >>>>>>>>>>> to specify the
> >>>>>>>>>>> valid time. So that you don't have to modify the config
> >>>>>>>>>>> file, I'd suggest
> >>>>>>>>>>> using environment variables:
> >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> >>>>>>>>>>>
> >>>>>>>>>>> fcst = {
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>> obs = {
> >>>>>>>>>>> convert(x) = x * 10^9;
> >>>>>>>>>>> field = [
> >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> >>>>>>>>>>> ];
> >>>>>>>>>>> }
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> >>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Sounds good, thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> John
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> >>>>>>>>>>>> Affiliate <
> >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids and
> >>>>>>>>>>>>> focus on
> >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
over
> >>>>>>>>>>>>> to Theia so we'll
> >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> >>>>>>>>>>>>> into problems, I'll
> >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
> >>>>>>>>>>>>> max/avg files).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not for
> >>>>>>>>>>>>>> bias
> >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
you
> >>>>>>>>>>>>>> may want to provide
> >>>>>>>>>>>>>> him para5 runs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> >>>>>>>>>>>>>> to do
> >>>>>>>>>>>>>> some testing some production files. Do you think I
> >>>>>>>>>>>>>> should use parallels
> >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> >>>>>>>>>>>>>> creating a MET+ use case
> >>>>>>>>>>>>>> for AQ?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> >>>>>>>>>>>>>> should
> >>>>>>>>>>>>>> be OK for now.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
We
> >>>>>>>>>>>>>>> wanted to update the error in daily max hour labels
but
> >>>>>>>>>>>>>>> management did not
> >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> >>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA Affiliate
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John over
> >>>>>>>>>>>>>>> parallel CMAQ data
> >>>>>>>>>>>>>>> rather than production CMAQ.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don't think that the MET team has any other
versions
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they can
> >>>>>>>>>>>>>>> also clarify this as
> >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does over
the
> >>>>>>>>>>>>>>> current wgrib2?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to use
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> latest parallel run output (para5).
> >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall Jianping
> >>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
did
> >>>>>>>>>>>>>>>> not want to risk any
> >>>>>>>>>>>>>>>> further delays so we will have to do so with the
FYQ4
> >>>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>> Jeff
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA Affiliate
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
think
> >>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> >>>>>>>>>>>>>>>> but I don't know if I
> >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I think
> >>>>>>>>>>>>>>>> the parallel CMAQ they
> >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
sure.
> >>>>>>>>>>>>>>>> But he would have
> >>>>>>>>>>>>>>>> more insight as to what is correct and how to
approach
> >>>>>>>>>>>>>>>> this.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
> >>>>>>>>>>>>>>>> talks
> >>>>>>>>>>>>>>>> about?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Perry
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley Gotway
<
> >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Perry,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
compute
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
hour
> >>>>>>>>>>>>>>>>> ozone concentrations.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>> pb2nc_output.nc
> >>>>>>>>>>>>>>>>> PB2NCConfig \
> >>>>>>>>>>>>>>>>> -pbfile aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
\
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
\
> >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Using the attached config files.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
FULL,
> >>>>>>>>>>>>>>>>> for interpolation
> >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But I noticed some oddities:
> >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
this
> >>>>>>>>>>>>>>>>> "max"
> >>>>>>>>>>>>>>>>> file:
> >>>>>>>>>>>>>>>>> wgrib2
> >>>>>>>>>>>>>>>>> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-40
> >>>>>>>>>>>>>>>>> hour
> >>>>>>>>>>>>>>>>> ave fcst:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
> >>>>>>>>>>>>>>>>> revealed
> >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17
> >>>>>>>>>>>>>>>>> and 41, respectively.
> >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the same
> >>>>>>>>>>>>>>>>> file, we
> >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which one
to
> >>>>>>>>>>>>>>>>> use. Do this by
> >>>>>>>>>>>>>>>>> defining the "lead_time":
> >>>>>>>>>>>>>>>>> fcst = {
> >>>>>>>>>>>>>>>>> field = [
> >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time =
> >>>>>>>>>>>>>>>>> "41";' }
> >>>>>>>>>>>>>>>>> ];
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off
> >>>>>>>>>>>>>>>>> by 1
> >>>>>>>>>>>>>>>>> hour.
> >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
time
> >>>>>>>>>>>>>>>>> =
> >>>>>>>>>>>>>>>>> valid time of 2018052805.
> >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max value
> >>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z while
> >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat config
> >>>>>>>>>>>>>>>>> file by
> >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> >>>>>>>>>>>>>>>>> obs_window = {
> >>>>>>>>>>>>>>>>> beg = -3600;
> >>>>>>>>>>>>>>>>> end = -3600;
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So what're the next steps?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We're making progress on the time summaries, but
I
> >>>>>>>>>>>>>>>>>> found
> >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
here:
> >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute 24-
hour
> >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> >>>>>>>>>>>>>>>>>> to an ascii file using
> >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to count
up
> >>>>>>>>>>>>>>>>>> the number of regular
> >>>>>>>>>>>>>>>>>> and maximum obs values:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
wc
> >>>>>>>>>>>>>>>>>> -l
> >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt |
> >>>>>>>>>>>>>>>>>> sort -u
> >>>>>>>>>>>>>>>>>> | wc -l
> >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt | wc
-l
> >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
sort
> >>>>>>>>>>>>>>>>>> -u |
> >>>>>>>>>>>>>>>>>> wc -l
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
all
> >>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
> >>>>>>>>>>>>>>>>>> meaning
> >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> >>>>>>>>>>>>>>>>>> duplicates.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> For example, the last station ID in that file is
> >>>>>>>>>>>>>>>>>> named
> >>>>>>>>>>>>>>>>>> "2FB31EEC":
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
> >>>>>>>>>>>>>>>>>> 2FB31EEC
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000 32.47
> >>>>>>>>>>>>>>>>>> -114.77
> >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in the
> >>>>>>>>>>>>>>>>>> last two
> >>>>>>>>>>>>>>>>>> lines.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> >>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> showing up?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
<hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Cheers.
> >>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
we'd
> >>>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across 2
> >>>>>>>>>>>>>>>>>>> files. Do we have existing
> >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
> >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> >>>>>>>>>>>>>>>>>>> temperature or something?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> >>>>>>>>>>>>>>>>>>>> sec).
> >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in the
> >>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content of
> >>>>>>>>>>>>>>>>>>>>> obs_hid
> >>>>>>>>>>>>>>>>>>>>> variable).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
> >>>>>>>>>>>>>>>>>>>>> (test
> >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
> >>>>>>>>>>>>>>>>>>>>> added one
> >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change the
> >>>>>>>>>>>>>>>>>>>>> nightly
> >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
> >>>>>>>>>>>>>>>>>>>>>> data.
> >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The output is available
> >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
> >>>>>>>>>>>>>>>>>>>>>> Gotway <
> >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic for
> >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr counts:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
> >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to dump
to
> >>>>>>>>>>>>>>>>>>>>>>> ascii
> >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000 string
> >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC not
> >>>>>>>>>>>>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
> >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John Halley
> >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard Soh
<
> >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
> >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> >>>>>>>>>>>>>>>>>>>>>>>>> from aqm.20180528/aqm.t12z.prepbufr.tm00.
> >>>>>>>>>>>>>>>>>>>>>>>>> There
> >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
I'm
> >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
take
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC. I
> >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >>>>>>>>>>>>>>>>>>>>>>>>>> \
> >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
> >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end = 0;
}
> >>>>>>>>>>>>>>>>>>>>>>>>>> //
> >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
using
> >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what you'd
> >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the "AIRNOW_MAX_240000"
> >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused by
> >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
Halley
> >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW PrepBUFR
> >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
hour
> >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is reasonable
for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
than
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295 currently)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for summary)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable. The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180622
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript /d1/johnhg/MET/MET_development/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all of
the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
them
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table pb2nc_output.nc
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: David Fillmore
Time: Thu May 02 15:43:11 2019
Hi Perry -
Are you running on theia?
John suggests that we try to troubleshoot there ...
thanks,
David
On Thu May 02 15:35:10 2019, fillmore wrote:
> Hi Perry -
>
> An update on where we are on this issue:
> Using the prepda.2019042312.nc file you provided we indeed reproduce
> the point_stat error.
> The point_stat error is due to only a single entry the valid time
> table,
> e.g. from ncdump -v hdr_vld_table
> hdr_vld_table =
> "20190423_050000",
> while the hdr_vld indices in that file are hdr_vld = 0, 33, 33, ...
> which are all out of range except the first.
>
> If I run pb2nc here with your PREPBUFR data and your PB2NCConfig_aqm
> config file, however,
> I generate a different prepda.2019042312.nc file that runs through
to
> completion with point_stat.
> Its valid time table:
>
> hdr_vld_table =
> "20190423_030000",
> "20190423_010000",
> "20190423_020000",
> "20190423_230000",
> "20190423_150000",
> "20190423_090000",
> "20190423_070000",
> "20190423_060000",
> "20190423_210000",
> "20190423_050000",
> "20190423_170000",
> "20190423_100000",
> "20190423_200000",
> "20190423_140000",
> "20190423_130000",
> "20190423_160000",
> "20190423_040000",
> "20190423_180000",
> "20190423_080000",
> "20190423_120000",
> "20190423_110000",
> "20190423_220000",
> "20190423_190000",
> "20190424_000000" ;
> }
>
> So it seems pb2nc is running correctly here, but not on your side.
> Let me discuss with John the best way to proceed ...
> thanks,
> David
>
>
> On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > Hi, everyone,
> >
> > I am attaching this to this long thread regarding the daily max of
> > ozone
> > verification, so we remember what we were doing back then.
> >
> > I am now running this again using MET 8.0, and something seems to
not
> > be
> > working when I run point_stat. I get this error when running a
daily
> > max
> > of ozone:
> >
> > ERROR :
> > ERROR : StringArray::operator[](int) const -> range check error!
> > ERROR :
> >
> > I am using my own scripts to run this using MET, so this is not
using
> > METplus.
> >
> > You can find a log file of this here:
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > It
> > runs pb2nc and point_stat for max ozone. The pb2nc code is working
> > fine, I
> > believe.
> >
> > The script I am using to run this is here:
> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> >
> > The prepbufr files are:
> >
> > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> >
> > The model file is here:
> >
> >
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> >
> >
> > What could be causing this error?
> >
> > Thanks!
> >
> > Perry
> >
> >
> > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Great, glad that did the trick.
> > >
> > > John
> > >
> > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate <
> > > perry.shafran at noaa.gov> wrote:
> > >
> > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-hr
> > >> avg.
> > >>
> > >> Thanks!
> > >>
> > >> Perry
> > >>
> > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > >> <johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >>> Perry,
> > >>>
> > >>> I think the problem is in your setting of
> > >>> valid_time="${CUR_VX_TIME}";
> > >>>
> > >>> Here's your wgrib2 output:
> > >>> wgrib2 oz_DAY1
> > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
> > >>> fcst:
> > >>>
> > >>> The "d=2018081112" tells you the model initialization time.
The
> > >>> valid
> > >>> time = init time + forecast time. So the OZMAX1 data is valid
16
> > >>> hours
> > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> > >>> wgrib2
> > >>> command using the "-vt" argument to list the verification
time.
> > >>>
> > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > >>> valid
> > >>> at the time you're requesting.
> > >>>
> > >>> John
> > >>>
> > >>>
> > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA Affiliate
<
> > >>> perry.shafran at noaa.gov> wrote:
> > >>>
> > >>>> Hi, John,
> > >>>>
> > >>>> I haven't been able to work on this for awhile, but now I'm
> > >>>> working on
> > >>>> it again but seem to be running into issues.
> > >>>>
> > >>>> I have a file that has both the daily max for 1-hr ozone and
the
> > >>>> daily
> > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > >>>>
> > >>>> 9 (t10a2)
> > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>> >
> > >>>> wgrib2 oz_DAY1
> > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > >>>> 10 (t10a2)
> > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>> >
> > >>>>
> > >>>> As you can see they have different forecast times. But when
I
> > >>>> run
> > >>>> point_stat, I get this output:
> > >>>>
> > >>>> DEBUG 2:
> > >>>> DEBUG 2:
> > >>>>
--------------------------------------------------------------------------------
> > >>>> DEBUG 2:
> > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > >>>> WARNING:
> > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> > >>>> records
> > >>>> found for 'OZMAX8/L1'
> > >>>> WARNING:
> > >>>> WARNING:
> > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> > >>>> OZMAX8/L1
> > >>>> found in file: oz_DAY1
> > >>>> WARNING:
> > >>>>
> > >>>> Here is the relevant portion of the point_stat config file:
> > >>>>
> > >>>> fcst = {
> > >>>> field = [
> > >>>> { name = "OZMAX1"; level = "L1";
> > >>>> valid_time="${CUR_VX_TIME}"; },
> > >>>> { name = "OZMAX8"; level = "L1";
> > >>>> valid_time="${CUR_VX_TIME}"; }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> obs = {
> > >>>> convert(x) = x * 10^9;
> > >>>> field = [
> > >>>> { name = "COPO"; level = "A1"; message_type =
> > >>>> "AIRNOW_MAX_230000"; },
> > >>>> { name = "COPO"; level = "A8"; message_type =
> > >>>> "AIRNOW_MAX_230000"; }
> > >>>> ];
> > >>>> }
> > >>>>
> > >>>> I don't remember if I had this working previously for both
the
> > >>>> 1hr
> > >>>> and
> > >>>> the 8hr ozone, but apparently there is some sort of issue for
> > >>>> the
> > >>>> 8hr
> > >>>> ozone. Have any thoughts?
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>> Perry
> > >>>>
> > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate
> > >>>> <
> > >>>> perry.shafran at noaa.gov> wrote:
> > >>>>
> > >>>>> I chose option 2, and that seems to work - I have a .stat
file
> > >>>>> with
> > >>>>> output. Now I'll need to do some work here to see if the
> > >>>>> output
> > >>>>> looks OK,
> > >>>>> and to configure the verification so it verifies OZMAX1 and
> > >>>>> OZMAX8.
> > >>>>>
> > >>>>> I also wonder if this would be easily modified for PM.
> > >>>>>
> > >>>>> Perry
> > >>>>>
> > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > >>>>> <johnhg at ucar.edu>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Perry,
> > >>>>>>
> > >>>>>> Yep, you've configured/run PB2NC to compute time summaries
> > >>>>>> that
> > >>>>>> are
> > >>>>>> valid at 04Z. And you're comparing them to forecasts that
are
> > >>>>>> purportedly
> > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > >>>>>> rejected due
> > >>>>>> to the valid time.
> > >>>>>>
> > >>>>>> It's up to you guys to decide what the "correct" way of
> > >>>>>> handling
> > >>>>>> this
> > >>>>>> is... but here are a couple of options:
> > >>>>>>
> > >>>>>> (1) You could use the existing setup and address this logic
in
> > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > >>>>>> from 0 to:
> > >>>>>>
> > >>>>>> obs_window = {
> > >>>>>> beg = 3600;
> > >>>>>> end = 3600;
> > >>>>>> }
> > >>>>>> This tells Point-Stat to match observations which are
exactly
> > >>>>>> 1-
> > >>>>>> hour
> > >>>>>> prior to the valid time of the forecast.
> > >>>>>>
> > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z
> > >>>>>> to
> > >>>>>> 05Z
> > >>>>>> instead and then leave the Point-Stat obs_window setting at
0:
> > >>>>>>
> > >>>>>> time_summary = {
> > >>>>>> flag = TRUE;
> > >>>>>> beg = "050000";
> > >>>>>> end = "050000";
> > >>>>>> ...
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> John
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > >>>>>> Affiliate
> > >>>>>> <
> > >>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>
> > >>>>>>> Oops, spoke too soon. There *is* a .stat output file, but
> > >>>>>>> it's
> > >>>>>>> empty. Here's the output from point_stat:
> > >>>>>>>
> > >>>>>>> DEBUG 4:
> > >>>>>>> DEBUG 4: Data plane information:
> > >>>>>>> DEBUG 4: plane min: 13.86
> > >>>>>>> DEBUG 4: plane max: 130.82
> > >>>>>>> DEBUG 4: scan mode: 64
> > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > >>>>>>> DEBUG 4: lead time: 170000
> > >>>>>>> DEBUG 4: init time: 20180708_120000
> > >>>>>>> DEBUG 4: bitmap flag: 255
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > >>>>>>> type
> > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > >>>>>>> NEAREST(1),
> > >>>>>>> using 0 pairs.
> > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > >>>>>>> DEBUG 3: Observations processed = 124637
> > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > >>>>>>>
> > >>>>>>> Looks like most rejections were due to valid time.
> > >>>>>>>
> > >>>>>>> Have any thoughts about this one?
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>>
> > >>>>>>> Perry
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> > >>>>>>> Affiliate
> > >>>>>>> <
> > >>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>
> > >>>>>>>> Hi, John,
> > >>>>>>>>
> > >>>>>>>> Looks like the changing of the time stamp from _04 to _05
> > >>>>>>>> did
> > >>>>>>>> the
> > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > >>>>>>>>
> > >>>>>>>> Thanks!
> > >>>>>>>>
> > >>>>>>>> Perry
> > >>>>>>>>
> > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > >>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>
> > >>>>>>>>> Perry,
> > >>>>>>>>>
> > >>>>>>>>> I agree, there's likely some issue with requesting the
> > >>>>>>>>> timestamp
> > >>>>>>>>> of the data from that file. I think switching the
> > >>>>>>>>> verification time to
> > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > >>>>>>>>>
> > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > >>>>>>>>> times
> > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > >>>>>>>>> and
> > >>>>>>>>> MET commands
> > >>>>>>>>> to illustrate.
> > >>>>>>>>>
> > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t), forecast
> > >>>>>>>>> time
> > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> > >>>>>>>>> data file you
> > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> > >>>>>>>>> so
> > >>>>>>>>> I have that
> > >>>>>>>>> funky forecast hour:
> > >>>>>>>>>
> > >>>>>>>>> wgrib2 -t -ftime -vt
> > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > >>>>>>>>> fcst:vt=2018070505
> > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave fcst:vt=2018070605
> > >>>>>>>>>
> > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
> > >>>>>>>>> level
> > >>>>>>>>> (-v
> > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > >>>>>>>>> reading:
> > >>>>>>>>>
> > >>>>>>>>> # Read the first record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > >>>>>>>>> DEBUG 4: lead time: 230000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>> ...
> > >>>>>>>>> # Read the second record
> > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > >>>>>>>>> ...
> > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > >>>>>>>>> DEBUG 4: lead time: 470000
> > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > >>>>>>>>>
> > >>>>>>>>> Hope that helps.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> John
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> > >>>>>>>>> Affiliate <
> > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, John,
> > >>>>>>>>>>
> > >>>>>>>>>> When I implemented all the changes (and also set it up
so
> > >>>>>>>>>> it's
> > >>>>>>>>>> reading the beta6 version), I used this command:
> > >>>>>>>>>>
> > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > >>>>>>>>>>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > >>>>>>>>>> -v 4
> > >>>>>>>>>>
> > >>>>>>>>>> But then I got this error:
> > >>>>>>>>>>
> > >>>>>>>>>> DEBUG 2:
> > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> > >>>>>>>>>> matching
> > >>>>>>>>>> records found for 'OZMAX1/L1'
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > >>>>>>>>>> WARNING:
> > >>>>>>>>>> ERROR :
> > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
> > >>>>>>>>>> forecast
> > >>>>>>>>>> data
> > >>>>>>>>>> found! Exiting...
> > >>>>>>>>>>
> > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output for
the
> > >>>>>>>>>> file
> > >>>>>>>>>> in this case reads:
> > >>>>>>>>>>
> > >>>>>>>>>> 14 (t14a2)
> > >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>
> > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > >>>>>>>>>> 15 (t14a2)
> > >>>>>>>>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >>>>>>>>>>
> > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> > >>>>>>>>>> suggested
> > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> > >>>>>>>>>> it needs to be
> > >>>>>>>>>> the time stamp of the first day?
> > >>>>>>>>>>
> > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for any thoughts.
> > >>>>>>>>>>
> > >>>>>>>>>> Perry
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Perry,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm confused. In the previous air quality vx we did,
we
> > >>>>>>>>>>> used
> > >>>>>>>>>>> observation files named:
> > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > >>>>>>>>>>>
> > >>>>>>>>>>> And those files don't exist in the July data you just
> > >>>>>>>>>>> sent.
> > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm not sure if you're actually verifying against the
> > >>>>>>>>>>> BUFR
> > >>>>>>>>>>> or
> > >>>>>>>>>>> PREPBUFR.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > >>>>>>>>>>> settings...
> > >>>>>>>>>>>
> > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
> > >>>>>>>>>>> files
> > >>>>>>>>>>> -
> > >>>>>>>>>>> thecurrent day and the previous day...
> > >>>>>>>>>>> pb2nc \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > >>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>
> > >>>>>>>>>>> (2) In the PB2NC config file, set up the time_summary
> > >>>>>>>>>>> like
> > >>>>>>>>>>> this:
> > >>>>>>>>>>> time_summary = {
> > >>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>> end = "040000";
> > >>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
> > >>>>>>>>>>> previous
> > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > >>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > >>>>>>>>>>> type = [ "max", "mean" ];
> > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries when
at
> > >>>>>>>>>>> least
> > >>>>>>>>>>> 75% of expect obs are present
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > >>>>>>>>>>> over the
> > >>>>>>>>>>> defined time period. They're identified in the output
by
> > >>>>>>>>>>> the message type:
> > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > >>>>>>>>>>>
> > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > >>>>>>>>>>> sections like this for the max of the 1-hour data.
> > >>>>>>>>>>> Instead
> > >>>>>>>>>>> of specifying
> > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> > >>>>>>>>>>> to specify the
> > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > >>>>>>>>>>> file, I'd suggest
> > >>>>>>>>>>> using environment variables:
> > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > >>>>>>>>>>>
> > >>>>>>>>>>> fcst = {
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>> obs = {
> > >>>>>>>>>>> convert(x) = x * 10^9;
> > >>>>>>>>>>> field = [
> > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > >>>>>>>>>>> ];
> > >>>>>>>>>>> }
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Sounds good, thanks!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> John
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran - NOAA
> > >>>>>>>>>>>> Affiliate <
> > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and
> > >>>>>>>>>>>>> focus on
> > >>>>>>>>>>>>> the non-bc files. It takes some time to copy files
> > >>>>>>>>>>>>> over
> > >>>>>>>>>>>>> to Theia so we'll
> > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> > >>>>>>>>>>>>> into problems, I'll
> > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-hr
> > >>>>>>>>>>>>> max/avg files).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for
> > >>>>>>>>>>>>>> bias
> > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be safe,
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>> may want to provide
> > >>>>>>>>>>>>>> him para5 runs.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA Affiliate
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> > >>>>>>>>>>>>>> to do
> > >>>>>>>>>>>>>> some testing some production files. Do you think I
> > >>>>>>>>>>>>>> should use parallels
> > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> > >>>>>>>>>>>>>> creating a MET+ use case
> > >>>>>>>>>>>>>> for AQ?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > >>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>> be OK for now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Actually, I think I was wrong about the parallels.
> > >>>>>>>>>>>>>>> We
> > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > >>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>> management did not
> > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ upgrade.
> > >>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > >>>>>>>>>>>>>>> parallel CMAQ data
> > >>>>>>>>>>>>>>> rather than production CMAQ.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I don't think that the MET team has any other
> > >>>>>>>>>>>>>>> versions
> > >>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can
> > >>>>>>>>>>>>>>> also clarify this as
> > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> current wgrib2?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > >>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but management
> > >>>>>>>>>>>>>>>> did
> > >>>>>>>>>>>>>>>> not want to risk any
> > >>>>>>>>>>>>>>>> further delays so we will have to do so with the
> > >>>>>>>>>>>>>>>> FYQ4
> > >>>>>>>>>>>>>>>> implementation.
> > >>>>>>>>>>>>>>>> Jeff
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
> > >>>>>>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> > >>>>>>>>>>>>>>>> but I don't know if I
> > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
> > >>>>>>>>>>>>>>>> sure.
> > >>>>>>>>>>>>>>>> But he would have
> > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
> > >>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>> this.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues John
> > >>>>>>>>>>>>>>>> talks
> > >>>>>>>>>>>>>>>> about?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Perry
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Perry,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
> > >>>>>>>>>>>>>>>>> compute
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and 8-
> > >>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>> ozone concentrations.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Using the attached config files.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over region
> > >>>>>>>>>>>>>>>>> FULL,
> > >>>>>>>>>>>>>>>>> for interpolation
> > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents of
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> "max"
> > >>>>>>>>>>>>>>>>> file:
> > >>>>>>>>>>>>>>>>> wgrib2
> > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma level:17-
40
> > >>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>> ave fcst:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But testing
> > >>>>>>>>>>>>>>>>> revealed
> > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast hours
17
> > >>>>>>>>>>>>>>>>> and 41, respectively.
> > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded better.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> > >>>>>>>>>>>>>>>>> file, we
> > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> use. Do this by
> > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > >>>>>>>>>>>>>>>>> fcst = {
> > >>>>>>>>>>>>>>>>> field = [
> > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time
> > >>>>>>>>>>>>>>>>> =
> > >>>>>>>>>>>>>>>>> "41";' }
> > >>>>>>>>>>>>>>>>> ];
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps are
off
> > >>>>>>>>>>>>>>>>> by 1
> > >>>>>>>>>>>>>>>>> hour.
> > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour lead
> > >>>>>>>>>>>>>>>>> time
> > >>>>>>>>>>>>>>>>> =
> > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> > >>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> > >>>>>>>>>>>>>>>>> file by
> > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > >>>>>>>>>>>>>>>>> obs_window = {
> > >>>>>>>>>>>>>>>>> beg = -3600;
> > >>>>>>>>>>>>>>>>> end = -3600;
> > >>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> So what're the next steps?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> It was caused by two input files and was fixed.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> > >>>>>>>>>>>>>>>>>> found
> > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
> > >>>>>>>>>>>>>>>>>> here:
> > >>>>>>>>>>>>>>>>>> /d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-
> > >>>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> > >>>>>>>>>>>>>>>>>> to an ascii file using
> > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count
> > >>>>>>>>>>>>>>>>>> up
> > >>>>>>>>>>>>>>>>>> the number of regular
> > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
|
> > >>>>>>>>>>>>>>>>>> wc
> > >>>>>>>>>>>>>>>>>> -l
> > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000 pb2nc_output.txt
|
> > >>>>>>>>>>>>>>>>>> sort -u
> > >>>>>>>>>>>>>>>>>> | wc -l
> > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
wc
> > >>>>>>>>>>>>>>>>>> -l
> > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt |
> > >>>>>>>>>>>>>>>>>> sort
> > >>>>>>>>>>>>>>>>>> -u |
> > >>>>>>>>>>>>>>>>>> wc -l
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning that
> > >>>>>>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns 6,555
> > >>>>>>>>>>>>>>>>>> meaning
> > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > >>>>>>>>>>>>>>>>>> duplicates.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is
> > >>>>>>>>>>>>>>>>>> named
> > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt | grep
> > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > >>>>>>>>>>>>>>>>>> -114.77
> > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the
> > >>>>>>>>>>>>>>>>>> last two
> > >>>>>>>>>>>>>>>>>> lines.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> > >>>>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>> showing up?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Cheers.
> > >>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason why
> > >>>>>>>>>>>>>>>>>>> we'd
> > >>>>>>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values across
2
> > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we could
> > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > >>>>>>>>>>>>>>>>>>> temperature or something?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > >>>>>>>>>>>>>>>>>>>> sec).
> > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the
> > >>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the content
of
> > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > >>>>>>>>>>>>>>>>>>>>> variable).
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time summary
> > >>>>>>>>>>>>>>>>>>>>> (test
> > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs. I
> > >>>>>>>>>>>>>>>>>>>>> added one
> > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> > >>>>>>>>>>>>>>>>>>>>> nightly
> > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not currently
> > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh <
> > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time summary
> > >>>>>>>>>>>>>>>>>>>>>> data.
> > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John Halley
> > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute 24-
hour
> > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
> > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > >>>>>>>>>>>>>>>>>>>>>>> actually
> > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in the
> > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00 and
> > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs data.
> > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you please
> > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running PB2NC.
I
> > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > >>>>>>>>>>>>>>>>>>>>>>>>>> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-hourly
> > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end =
0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
> > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of them
to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a 24
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE headers
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0 to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on dakota
in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180622
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced by
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or table
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri May 03 06:35:41 2019
Hi, David, John,
I am not running on Theia. In fact I think it would take quite a bit
of
effort to transition this thing to Theia, and I'll need to be running
this
verification on prod data anyway, which is only on WCOSS.
However, if you wish to troubleshoot on Theia, you should do so, and I
think you have all you need to be able to do so, right?
It would seem that the error on my side is in PB2NC, so I'll probably
need
to make sure I'm running what you're running.
Thanks!
Perry
On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT
<met_help at ucar.edu>
wrote:
> Hi Perry -
> Are you running on theia?
> John suggests that we try to troubleshoot there ...
> thanks,
> David
>
> On Thu May 02 15:35:10 2019, fillmore wrote:
> > Hi Perry -
> >
> > An update on where we are on this issue:
> > Using the prepda.2019042312.nc file you provided we indeed
reproduce
> > the point_stat error.
> > The point_stat error is due to only a single entry the valid time
> > table,
> > e.g. from ncdump -v hdr_vld_table
> > hdr_vld_table =
> > "20190423_050000",
> > while the hdr_vld indices in that file are hdr_vld = 0, 33, 33,
...
> > which are all out of range except the first.
> >
> > If I run pb2nc here with your PREPBUFR data and your
PB2NCConfig_aqm
> > config file, however,
> > I generate a different prepda.2019042312.nc file that runs through
to
> > completion with point_stat.
> > Its valid time table:
> >
> > hdr_vld_table =
> > "20190423_030000",
> > "20190423_010000",
> > "20190423_020000",
> > "20190423_230000",
> > "20190423_150000",
> > "20190423_090000",
> > "20190423_070000",
> > "20190423_060000",
> > "20190423_210000",
> > "20190423_050000",
> > "20190423_170000",
> > "20190423_100000",
> > "20190423_200000",
> > "20190423_140000",
> > "20190423_130000",
> > "20190423_160000",
> > "20190423_040000",
> > "20190423_180000",
> > "20190423_080000",
> > "20190423_120000",
> > "20190423_110000",
> > "20190423_220000",
> > "20190423_190000",
> > "20190424_000000" ;
> > }
> >
> > So it seems pb2nc is running correctly here, but not on your side.
> > Let me discuss with John the best way to proceed ...
> > thanks,
> > David
> >
> >
> > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > Hi, everyone,
> > >
> > > I am attaching this to this long thread regarding the daily max
of
> > > ozone
> > > verification, so we remember what we were doing back then.
> > >
> > > I am now running this again using MET 8.0, and something seems
to not
> > > be
> > > working when I run point_stat. I get this error when running a
daily
> > > max
> > > of ozone:
> > >
> > > ERROR :
> > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > ERROR :
> > >
> > > I am using my own scripts to run this using MET, so this is not
using
> > > METplus.
> > >
> > > You can find a log file of this here:
> > >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > It
> > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> > > fine, I
> > > believe.
> > >
> > > The script I am using to run this is here:
> > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > >
> > > The prepbufr files are:
> > >
> > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > >
> > > The model file is here:
> > >
> > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > >
> > >
> > > What could be causing this error?
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > >
> > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Great, glad that did the trick.
> > > >
> > > > John
> > > >
> > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA Affiliate
<
> > > > perry.shafran at noaa.gov> wrote:
> > > >
> > > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the 1-
hr
> > > >> avg.
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Perry
> > > >>
> > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > >> <johnhg at ucar.edu>
> > > >> wrote:
> > > >>
> > > >>> Perry,
> > > >>>
> > > >>> I think the problem is in your setting of
> > > >>> valid_time="${CUR_VX_TIME}";
> > > >>>
> > > >>> Here's your wgrib2 output:
> > > >>> wgrib2 oz_DAY1
> > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
> > > >>> fcst:
> > > >>>
> > > >>> The "d=2018081112" tells you the model initialization time.
The
> > > >>> valid
> > > >>> time = init time + forecast time. So the OZMAX1 data is
valid 16
> > > >>> hours
> > > >>> later while the OZMAX8 data is valid 22 hours later. Rerun
that
> > > >>> wgrib2
> > > >>> command using the "-vt" argument to list the verification
time.
> > > >>>
> > > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > > >>> valid
> > > >>> at the time you're requesting.
> > > >>>
> > > >>> John
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate <
> > > >>> perry.shafran at noaa.gov> wrote:
> > > >>>
> > > >>>> Hi, John,
> > > >>>>
> > > >>>> I haven't been able to work on this for awhile, but now I'm
> > > >>>> working on
> > > >>>> it again but seem to be running into issues.
> > > >>>>
> > > >>>> I have a file that has both the daily max for 1-hr ozone
and the
> > > >>>> daily
> > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > >>>>
> > > >>>> 9 (t10a2)
> > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>> >
> > > >>>> wgrib2 oz_DAY1
> > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave fcst:
> > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > >>>> 10 (t10a2)
> > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>> >
> > > >>>>
> > > >>>> As you can see they have different forecast times. But
when I
> > > >>>> run
> > > >>>> point_stat, I get this output:
> > > >>>>
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2:
> > > >>>>
>
--------------------------------------------------------------------------------
> > > >>>> DEBUG 2:
> > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > >>>> WARNING:
> > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No matching
> > > >>>> records
> > > >>>> found for 'OZMAX8/L1'
> > > >>>> WARNING:
> > > >>>> WARNING:
> > > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> > > >>>> OZMAX8/L1
> > > >>>> found in file: oz_DAY1
> > > >>>> WARNING:
> > > >>>>
> > > >>>> Here is the relevant portion of the point_stat config file:
> > > >>>>
> > > >>>> fcst = {
> > > >>>> field = [
> > > >>>> { name = "OZMAX1"; level = "L1";
> > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > >>>> { name = "OZMAX8"; level = "L1";
> > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> obs = {
> > > >>>> convert(x) = x * 10^9;
> > > >>>> field = [
> > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; },
> > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > >>>> "AIRNOW_MAX_230000"; }
> > > >>>> ];
> > > >>>> }
> > > >>>>
> > > >>>> I don't remember if I had this working previously for both
the
> > > >>>> 1hr
> > > >>>> and
> > > >>>> the 8hr ozone, but apparently there is some sort of issue
for
> > > >>>> the
> > > >>>> 8hr
> > > >>>> ozone. Have any thoughts?
> > > >>>>
> > > >>>> Thanks!
> > > >>>>
> > > >>>> Perry
> > > >>>>
> > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate
> > > >>>> <
> > > >>>> perry.shafran at noaa.gov> wrote:
> > > >>>>
> > > >>>>> I chose option 2, and that seems to work - I have a .stat
file
> > > >>>>> with
> > > >>>>> output. Now I'll need to do some work here to see if the
> > > >>>>> output
> > > >>>>> looks OK,
> > > >>>>> and to configure the verification so it verifies OZMAX1
and
> > > >>>>> OZMAX8.
> > > >>>>>
> > > >>>>> I also wonder if this would be easily modified for PM.
> > > >>>>>
> > > >>>>> Perry
> > > >>>>>
> > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > >>>>> <johnhg at ucar.edu>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Perry,
> > > >>>>>>
> > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > >>>>>> that
> > > >>>>>> are
> > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that are
> > > >>>>>> purportedly
> > > >>>>>> valid at 05Z. And that's the reason why you have so many
obs
> > > >>>>>> rejected due
> > > >>>>>> to the valid time.
> > > >>>>>>
> > > >>>>>> It's up to you guys to decide what the "correct" way of
> > > >>>>>> handling
> > > >>>>>> this
> > > >>>>>> is... but here are a couple of options:
> > > >>>>>>
> > > >>>>>> (1) You could use the existing setup and address this
logic in
> > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > > >>>>>> from 0 to:
> > > >>>>>>
> > > >>>>>> obs_window = {
> > > >>>>>> beg = 3600;
> > > >>>>>> end = 3600;
> > > >>>>>> }
> > > >>>>>> This tells Point-Stat to match observations which are
exactly
> > > >>>>>> 1-
> > > >>>>>> hour
> > > >>>>>> prior to the valid time of the forecast.
> > > >>>>>>
> > > >>>>>> (2) You could rerun PB2NC to compute a time summary from
05Z
> > > >>>>>> to
> > > >>>>>> 05Z
> > > >>>>>> instead and then leave the Point-Stat obs_window setting
at 0:
> > > >>>>>>
> > > >>>>>> time_summary = {
> > > >>>>>> flag = TRUE;
> > > >>>>>> beg = "050000";
> > > >>>>>> end = "050000";
> > > >>>>>> ...
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> John
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > > >>>>>> Affiliate
> > > >>>>>> <
> > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>
> > > >>>>>>> Oops, spoke too soon. There *is* a .stat output file,
but
> > > >>>>>>> it's
> > > >>>>>>> empty. Here's the output from point_stat:
> > > >>>>>>>
> > > >>>>>>> DEBUG 4:
> > > >>>>>>> DEBUG 4: Data plane information:
> > > >>>>>>> DEBUG 4: plane min: 13.86
> > > >>>>>>> DEBUG 4: plane max: 130.82
> > > >>>>>>> DEBUG 4: scan mode: 64
> > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > >>>>>>> DEBUG 4: lead time: 170000
> > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > > >>>>>>> type
> > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > > >>>>>>> NEAREST(1),
> > > >>>>>>> using 0 pairs.
> > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > >>>>>>>
> > > >>>>>>> Looks like most rejections were due to valid time.
> > > >>>>>>>
> > > >>>>>>> Have any thoughts about this one?
> > > >>>>>>>
> > > >>>>>>> Thanks!
> > > >>>>>>>
> > > >>>>>>> Perry
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> > > >>>>>>> Affiliate
> > > >>>>>>> <
> > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi, John,
> > > >>>>>>>>
> > > >>>>>>>> Looks like the changing of the time stamp from _04 to
_05
> > > >>>>>>>> did
> > > >>>>>>>> the
> > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > >>>>>>>>
> > > >>>>>>>> Thanks!
> > > >>>>>>>>
> > > >>>>>>>> Perry
> > > >>>>>>>>
> > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway <
> > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Perry,
> > > >>>>>>>>>
> > > >>>>>>>>> I agree, there's likely some issue with requesting the
> > > >>>>>>>>> timestamp
> > > >>>>>>>>> of the data from that file. I think switching the
> > > >>>>>>>>> verification time to
> > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > >>>>>>>>>
> > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > > >>>>>>>>> times
> > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > > >>>>>>>>> and
> > > >>>>>>>>> MET commands
> > > >>>>>>>>> to illustrate.
> > > >>>>>>>>>
> > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > >>>>>>>>> time
> > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> > > >>>>>>>>> data file you
> > > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> > > >>>>>>>>> so
> > > >>>>>>>>> I have that
> > > >>>>>>>>> funky forecast hour:
> > > >>>>>>>>>
> > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > >>>>>>>>> fcst:vt=2018070505
> > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > >>>>>>>>>
> > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher verbosity
> > > >>>>>>>>> level
> > > >>>>>>>>> (-v
> > > >>>>>>>>> 4) to have MET tell you the timing info of the data it
is
> > > >>>>>>>>> reading:
> > > >>>>>>>>>
> > > >>>>>>>>> # Read the first record
> > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>> ...
> > > >>>>>>>>> # Read the second record
> > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > >>>>>>>>> ...
> > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > >>>>>>>>>
> > > >>>>>>>>> Hope that helps.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> John
> > > >>>>>>>>>
> > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> > > >>>>>>>>> Affiliate <
> > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi, John,
> > > >>>>>>>>>>
> > > >>>>>>>>>> When I implemented all the changes (and also set it
up so
> > > >>>>>>>>>> it's
> > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > >>>>>>>>>>
> > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > >>>>>>>>>>
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > >>>>>>>>>> -v 4
> > > >>>>>>>>>>
> > > >>>>>>>>>> But then I got this error:
> > > >>>>>>>>>>
> > > >>>>>>>>>> DEBUG 2:
> > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> > > >>>>>>>>>> matching
> > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > >>>>>>>>>> WARNING:
> > > >>>>>>>>>> ERROR :
> > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
> > > >>>>>>>>>> forecast
> > > >>>>>>>>>> data
> > > >>>>>>>>>> found! Exiting...
> > > >>>>>>>>>>
> > > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output
for the
> > > >>>>>>>>>> file
> > > >>>>>>>>>> in this case reads:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 14 (t14a2)
> > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > >>>>>>>>>> 15 (t14a2)
> > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >>>>>>>>>>
> > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> > > >>>>>>>>>> suggested
> > > >>>>>>>>>> that it should be the time stamp of the 2nd day, but
maybe
> > > >>>>>>>>>> it needs to be
> > > >>>>>>>>>> the time stamp of the first day?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for any thoughts.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Perry
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway <
> > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Perry,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did, we
> > > >>>>>>>>>>> used
> > > >>>>>>>>>>> observation files named:
> > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> And those files don't exist in the July data you
just
> > > >>>>>>>>>>> sent.
> > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d aqm.t12z.anowpm.pb.tm00
> > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I'm not sure if you're actually verifying against
the
> > > >>>>>>>>>>> BUFR
> > > >>>>>>>>>>> or
> > > >>>>>>>>>>> PREPBUFR.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > > >>>>>>>>>>> settings...
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2 input
> > > >>>>>>>>>>> files
> > > >>>>>>>>>>> -
> > > >>>>>>>>>>> thecurrent day and the previous day...
> > > >>>>>>>>>>> pb2nc \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > >>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > >>>>>>>>>>> like
> > > >>>>>>>>>>> this:
> > > >>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
> > > >>>>>>>>>>> previous
> > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > >>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every hour
> > > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries
when at
> > > >>>>>>>>>>> least
> > > >>>>>>>>>>> 75% of expect obs are present
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > > >>>>>>>>>>> over the
> > > >>>>>>>>>>> defined time period. They're identified in the
output by
> > > >>>>>>>>>>> the message type:
> > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst" and
"obs"
> > > >>>>>>>>>>> sections like this for the max of the 1-hour data.
> > > >>>>>>>>>>> Instead
> > > >>>>>>>>>>> of specifying
> > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> > > >>>>>>>>>>> to specify the
> > > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > > >>>>>>>>>>> file, I'd suggest
> > > >>>>>>>>>>> using environment variables:
> > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> fcst = {
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>> obs = {
> > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > >>>>>>>>>>> field = [
> > > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type =
> > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > >>>>>>>>>>> ];
> > > >>>>>>>>>>> }
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway <
> > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Sounds good, thanks!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> John
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > >>>>>>>>>>>> Affiliate <
> > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc grids
and
> > > >>>>>>>>>>>>> focus on
> > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files
> > > >>>>>>>>>>>>> over
> > > >>>>>>>>>>>>> to Theia so we'll
> > > >>>>>>>>>>>>> let him work on the files he has for now and if he
runs
> > > >>>>>>>>>>>>> into problems, I'll
> > > >>>>>>>>>>>>> provide him with the para5 runs (at least the 24-
hr
> > > >>>>>>>>>>>>> max/avg files).
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I think the ops are ok for conus raw grids...not
for
> > > >>>>>>>>>>>>>> bias
> > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe,
> > > >>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>> may want to provide
> > > >>>>>>>>>>>>>> him para5 runs.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia for
John
> > > >>>>>>>>>>>>>> to do
> > > >>>>>>>>>>>>>> some testing some production files. Do you think
I
> > > >>>>>>>>>>>>>> should use parallels
> > > >>>>>>>>>>>>>> instead or do you think the ops files are OK for
> > > >>>>>>>>>>>>>> creating a MET+ use case
> > > >>>>>>>>>>>>>> for AQ?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the ops
files
> > > >>>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>> be OK for now.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels.
> > > >>>>>>>>>>>>>>> We
> > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > > >>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>> management did not
> > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > >>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Do any of the current parallels have the correct
time
> > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send John
over
> > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I don't think that the MET team has any other
> > > >>>>>>>>>>>>>>> versions
> > > >>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but they
can
> > > >>>>>>>>>>>>>>> also clarify this as
> > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> current wgrib2?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen <
> > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want to
use
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > > >>>>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management
> > > >>>>>>>>>>>>>>>> did
> > > >>>>>>>>>>>>>>>> not want to risk any
> > > >>>>>>>>>>>>>>>> further delays so we will have to do so with
the
> > > >>>>>>>>>>>>>>>> FYQ4
> > > >>>>>>>>>>>>>>>> implementation.
> > > >>>>>>>>>>>>>>>> Jeff
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation. I
> > > >>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some time
ago,
> > > >>>>>>>>>>>>>>>> but I don't know if I
> > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > >>>>>>>>>>>>>>>> are running corrects that, but not too entirely
> > > >>>>>>>>>>>>>>>> sure.
> > > >>>>>>>>>>>>>>>> But he would have
> > > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
> > > >>>>>>>>>>>>>>>> approach
> > > >>>>>>>>>>>>>>>> this.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues
John
> > > >>>>>>>>>>>>>>>> talks
> > > >>>>>>>>>>>>>>>> about?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Perry
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
> > > >>>>>>>>>>>>>>>>> compute
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1- and
8-
> > > >>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat \
> > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > >>>>>>>>>>>>>>>>> FULL,
> > > >>>>>>>>>>>>>>>>> for interpolation
> > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents
of
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>> "max"
> > > >>>>>>>>>>>>>>>>> file:
> > > >>>>>>>>>>>>>>>>> wgrib2
> > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
level:17-40
> > > >>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>> ave fcst:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > > >>>>>>>>>>>>>>>>> revealed
> > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours 17
> > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> > > >>>>>>>>>>>>>>>>> file, we
> > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it which
one
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> use. Do this by
> > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > >>>>>>>>>>>>>>>>> fcst = {
> > > >>>>>>>>>>>>>>>>> field = [
> > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time
> > > >>>>>>>>>>>>>>>>> =
> > > >>>>>>>>>>>>>>>>> "41";' }
> > > >>>>>>>>>>>>>>>>> ];
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps
are off
> > > >>>>>>>>>>>>>>>>> by 1
> > > >>>>>>>>>>>>>>>>> hour.
> > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead
> > > >>>>>>>>>>>>>>>>> time
> > > >>>>>>>>>>>>>>>>> =
> > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> > > >>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> > > >>>>>>>>>>>>>>>>> file by
> > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > > >>>>>>>>>>>>>>>>> obs_window = {
> > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > >>>>>>>>>>>>>>>>> end = -3600;
> > > >>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> We're making progress on the time summaries,
but I
> > > >>>>>>>>>>>>>>>>>> found
> > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on dakota
> > > >>>>>>>>>>>>>>>>>> here:
> > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to compute
24-
> > > >>>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count
> > > >>>>>>>>>>>>>>>>>> up
> > > >>>>>>>>>>>>>>>>>> the number of regular
> > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > >>>>>>>>>>>>>>>>>> wc
> > > >>>>>>>>>>>>>>>>>> -l
> > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > >>>>>>>>>>>>>>>>>> sort -u
> > > >>>>>>>>>>>>>>>>>> | wc -l
> > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000 pb2nc_output.txt
| wc
> > > >>>>>>>>>>>>>>>>>> -l
> > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000 pb2nc_output.txt
|
> > > >>>>>>>>>>>>>>>>>> sort
> > > >>>>>>>>>>>>>>>>>> -u |
> > > >>>>>>>>>>>>>>>>>> wc -l
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning
that
> > > >>>>>>>>>>>>>>>>>> all
> > > >>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > > >>>>>>>>>>>>>>>>>> meaning
> > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > > >>>>>>>>>>>>>>>>>> duplicates.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that file
is
> > > >>>>>>>>>>>>>>>>>> named
> > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > >>>>>>>>>>>>>>>>>> -114.77
> > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated in
the
> > > >>>>>>>>>>>>>>>>>> last two
> > > >>>>>>>>>>>>>>>>>> lines.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> > > >>>>>>>>>>>>>>>>>> are
> > > >>>>>>>>>>>>>>>>>> showing up?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > >>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason
why
> > > >>>>>>>>>>>>>>>>>>> we'd
> > > >>>>>>>>>>>>>>>>>>> have
> > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh <
> > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > > >>>>>>>>>>>>>>>>>>>> sec).
> > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check in
the
> > > >>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content of
> > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> > > >>>>>>>>>>>>>>>>>>>>> (test
> > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF inputs.
I
> > > >>>>>>>>>>>>>>>>>>>>> added one
> > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files and
will
> > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will change
the
> > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard Soh
<
> > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > >>>>>>>>>>>>>>>>>>>>>> at /d3/personal/hsoh/data/pb2nc_summary/
> > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the logic
for
> > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute
24-hour
> > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832 currently)
> > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump
> > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in PB2NC
not
> > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in
the
> > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > >>>>>>>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM Howard
Soh <
> > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00
and
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC. I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600; end
= 0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations... but
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem caused
by
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of
them to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a
24
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next week.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM hsoh
<
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation data
in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc without
any
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the NetCDF
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from 0
to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ variable.
The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger than
3655.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180622
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc > pb2nc_output.txt
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists all
of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But some
of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced
by
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
>
>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Fri May 03 10:02:27 2019
Perry,
We can log on to wcoss and test there instead. Yes, pb2nc does seem
to be
the culprit. Can you point us the wcoss node you're using and the
working
directory there? We basically need to recreate your call to pb2nc to
try
to replicate the behavior.
Thanks,
John
On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, David, John,
>
> I am not running on Theia. In fact I think it would take quite a
bit of
> effort to transition this thing to Theia, and I'll need to be
running this
> verification on prod data anyway, which is only on WCOSS.
>
> However, if you wish to troubleshoot on Theia, you should do so, and
I
> think you have all you need to be able to do so, right?
>
> It would seem that the error on my side is in PB2NC, so I'll
probably need
> to make sure I'm running what you're running.
>
> Thanks!
>
> Perry
>
> On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Perry -
> > Are you running on theia?
> > John suggests that we try to troubleshoot there ...
> > thanks,
> > David
> >
> > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > Hi Perry -
> > >
> > > An update on where we are on this issue:
> > > Using the prepda.2019042312.nc file you provided we indeed
reproduce
> > > the point_stat error.
> > > The point_stat error is due to only a single entry the valid
time
> > > table,
> > > e.g. from ncdump -v hdr_vld_table
> > > hdr_vld_table =
> > > "20190423_050000",
> > > while the hdr_vld indices in that file are hdr_vld = 0, 33, 33,
...
> > > which are all out of range except the first.
> > >
> > > If I run pb2nc here with your PREPBUFR data and your
PB2NCConfig_aqm
> > > config file, however,
> > > I generate a different prepda.2019042312.nc file that runs
through to
> > > completion with point_stat.
> > > Its valid time table:
> > >
> > > hdr_vld_table =
> > > "20190423_030000",
> > > "20190423_010000",
> > > "20190423_020000",
> > > "20190423_230000",
> > > "20190423_150000",
> > > "20190423_090000",
> > > "20190423_070000",
> > > "20190423_060000",
> > > "20190423_210000",
> > > "20190423_050000",
> > > "20190423_170000",
> > > "20190423_100000",
> > > "20190423_200000",
> > > "20190423_140000",
> > > "20190423_130000",
> > > "20190423_160000",
> > > "20190423_040000",
> > > "20190423_180000",
> > > "20190423_080000",
> > > "20190423_120000",
> > > "20190423_110000",
> > > "20190423_220000",
> > > "20190423_190000",
> > > "20190424_000000" ;
> > > }
> > >
> > > So it seems pb2nc is running correctly here, but not on your
side.
> > > Let me discuss with John the best way to proceed ...
> > > thanks,
> > > David
> > >
> > >
> > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > Hi, everyone,
> > > >
> > > > I am attaching this to this long thread regarding the daily
max of
> > > > ozone
> > > > verification, so we remember what we were doing back then.
> > > >
> > > > I am now running this again using MET 8.0, and something seems
to not
> > > > be
> > > > working when I run point_stat. I get this error when running
a daily
> > > > max
> > > > of ozone:
> > > >
> > > > ERROR :
> > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > ERROR :
> > > >
> > > > I am using my own scripts to run this using MET, so this is
not using
> > > > METplus.
> > > >
> > > > You can find a log file of this here:
> > > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > It
> > > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> > > > fine, I
> > > > believe.
> > > >
> > > > The script I am using to run this is here:
> > > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > >
> > > > The prepbufr files are:
> > > >
> > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > >
> > > > The model file is here:
> > > >
> > > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > >
> > > >
> > > > What could be causing this error?
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > >
> > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<johnhg at ucar.edu>
> > > > wrote:
> > > >
> > > > > Great, glad that did the trick.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
Affiliate <
> > > > > perry.shafran at noaa.gov> wrote:
> > > > >
> > > > >> Well this worked! And Jeff confirmed to me that the 8-hr
average
> > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in the
1-hr
> > > > >> avg.
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> Perry
> > > > >>
> > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > >> <johnhg at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >>> Perry,
> > > > >>>
> > > > >>> I think the problem is in your setting of
> > > > >>> valid_time="${CUR_VX_TIME}";
> > > > >>>
> > > > >>> Here's your wgrib2 output:
> > > > >>> wgrib2 oz_DAY1
> > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> > > > >>> fcst:
> > > > >>>
> > > > >>> The "d=2018081112" tells you the model initialization
time. The
> > > > >>> valid
> > > > >>> time = init time + forecast time. So the OZMAX1 data is
valid 16
> > > > >>> hours
> > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun that
> > > > >>> wgrib2
> > > > >>> command using the "-vt" argument to list the verification
time.
> > > > >>>
> > > > >>> Point-Stat isn't finding any OZMAX8 data because that data
isn't
> > > > >>> valid
> > > > >>> at the time you're requesting.
> > > > >>>
> > > > >>> John
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate <
> > > > >>> perry.shafran at noaa.gov> wrote:
> > > > >>>
> > > > >>>> Hi, John,
> > > > >>>>
> > > > >>>> I haven't been able to work on this for awhile, but now
I'm
> > > > >>>> working on
> > > > >>>> it again but seem to be running into issues.
> > > > >>>>
> > > > >>>> I have a file that has both the daily max for 1-hr ozone
and the
> > > > >>>> daily
> > > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > > >>>>
> > > > >>>> 9 (t10a2)
> > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >>>> >
> > > > >>>> wgrib2 oz_DAY1
> > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour ave
fcst:
> > > > >>>> 10 (t10a2)
> > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >>>> >
> > > > >>>>
> > > > >>>> As you can see they have different forecast times. But
when I
> > > > >>>> run
> > > > >>>> point_stat, I get this output:
> > > > >>>>
> > > > >>>> DEBUG 2:
> > > > >>>> DEBUG 2:
> > > > >>>>
> >
>
--------------------------------------------------------------------------------
> > > > >>>> DEBUG 2:
> > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > >>>> WARNING:
> > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > > >>>> records
> > > > >>>> found for 'OZMAX8/L1'
> > > > >>>> WARNING:
> > > > >>>> WARNING:
> > > > >>>> WARNING: process_fcst_climo_files() -> no fields matching
> > > > >>>> OZMAX8/L1
> > > > >>>> found in file: oz_DAY1
> > > > >>>> WARNING:
> > > > >>>>
> > > > >>>> Here is the relevant portion of the point_stat config
file:
> > > > >>>>
> > > > >>>> fcst = {
> > > > >>>> field = [
> > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > >>>> ];
> > > > >>>> }
> > > > >>>>
> > > > >>>> obs = {
> > > > >>>> convert(x) = x * 10^9;
> > > > >>>> field = [
> > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > >>>> "AIRNOW_MAX_230000"; },
> > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > >>>> "AIRNOW_MAX_230000"; }
> > > > >>>> ];
> > > > >>>> }
> > > > >>>>
> > > > >>>> I don't remember if I had this working previously for
both the
> > > > >>>> 1hr
> > > > >>>> and
> > > > >>>> the 8hr ozone, but apparently there is some sort of issue
for
> > > > >>>> the
> > > > >>>> 8hr
> > > > >>>> ozone. Have any thoughts?
> > > > >>>>
> > > > >>>> Thanks!
> > > > >>>>
> > > > >>>> Perry
> > > > >>>>
> > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
Affiliate
> > > > >>>> <
> > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>
> > > > >>>>> I chose option 2, and that seems to work - I have a
.stat file
> > > > >>>>> with
> > > > >>>>> output. Now I'll need to do some work here to see if
the
> > > > >>>>> output
> > > > >>>>> looks OK,
> > > > >>>>> and to configure the verification so it verifies OZMAX1
and
> > > > >>>>> OZMAX8.
> > > > >>>>>
> > > > >>>>> I also wonder if this would be easily modified for PM.
> > > > >>>>>
> > > > >>>>> Perry
> > > > >>>>>
> > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > > >>>>> <johnhg at ucar.edu>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Perry,
> > > > >>>>>>
> > > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > > >>>>>> that
> > > > >>>>>> are
> > > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that are
> > > > >>>>>> purportedly
> > > > >>>>>> valid at 05Z. And that's the reason why you have so
many obs
> > > > >>>>>> rejected due
> > > > >>>>>> to the valid time.
> > > > >>>>>>
> > > > >>>>>> It's up to you guys to decide what the "correct" way of
> > > > >>>>>> handling
> > > > >>>>>> this
> > > > >>>>>> is... but here are a couple of options:
> > > > >>>>>>
> > > > >>>>>> (1) You could use the existing setup and address this
logic in
> > > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > > > >>>>>> from 0 to:
> > > > >>>>>>
> > > > >>>>>> obs_window = {
> > > > >>>>>> beg = 3600;
> > > > >>>>>> end = 3600;
> > > > >>>>>> }
> > > > >>>>>> This tells Point-Stat to match observations which are
exactly
> > > > >>>>>> 1-
> > > > >>>>>> hour
> > > > >>>>>> prior to the valid time of the forecast.
> > > > >>>>>>
> > > > >>>>>> (2) You could rerun PB2NC to compute a time summary
from 05Z
> > > > >>>>>> to
> > > > >>>>>> 05Z
> > > > >>>>>> instead and then leave the Point-Stat obs_window
setting at 0:
> > > > >>>>>>
> > > > >>>>>> time_summary = {
> > > > >>>>>> flag = TRUE;
> > > > >>>>>> beg = "050000";
> > > > >>>>>> end = "050000";
> > > > >>>>>> ...
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>> John
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > > > >>>>>> Affiliate
> > > > >>>>>> <
> > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>
> > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output file,
but
> > > > >>>>>>> it's
> > > > >>>>>>> empty. Here's the output from point_stat:
> > > > >>>>>>>
> > > > >>>>>>> DEBUG 4:
> > > > >>>>>>> DEBUG 4: Data plane information:
> > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
observation
> > > > >>>>>>> type
> > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for interpolation
method
> > > > >>>>>>> NEAREST(1),
> > > > >>>>>>> using 0 pairs.
> > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > >>>>>>>
> > > > >>>>>>> Looks like most rejections were due to valid time.
> > > > >>>>>>>
> > > > >>>>>>> Have any thoughts about this one?
> > > > >>>>>>>
> > > > >>>>>>> Thanks!
> > > > >>>>>>>
> > > > >>>>>>> Perry
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran - NOAA
> > > > >>>>>>> Affiliate
> > > > >>>>>>> <
> > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Hi, John,
> > > > >>>>>>>>
> > > > >>>>>>>> Looks like the changing of the time stamp from _04 to
_05
> > > > >>>>>>>> did
> > > > >>>>>>>> the
> > > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks!
> > > > >>>>>>>>
> > > > >>>>>>>> Perry
> > > > >>>>>>>>
> > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley Gotway
<
> > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Perry,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I agree, there's likely some issue with requesting
the
> > > > >>>>>>>>> timestamp
> > > > >>>>>>>>> of the data from that file. I think switching the
> > > > >>>>>>>>> verification time to
> > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
initialization
> > > > >>>>>>>>> times
> > > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > > > >>>>>>>>> and
> > > > >>>>>>>>> MET commands
> > > > >>>>>>>>> to illustrate.
> > > > >>>>>>>>>
> > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > > >>>>>>>>> time
> > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using a
sample
> > > > >>>>>>>>> data file you
> > > > >>>>>>>>> sent to me. But remember I'm still using the "old"
wgrib2
> > > > >>>>>>>>> so
> > > > >>>>>>>>> I have that
> > > > >>>>>>>>> funky forecast hour:
> > > > >>>>>>>>>
> > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > > >>>>>>>>> fcst:vt=2018070505
> > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > > >>>>>>>>>
> > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
verbosity
> > > > >>>>>>>>> level
> > > > >>>>>>>>> (-v
> > > > >>>>>>>>> 4) to have MET tell you the timing info of the data
it is
> > > > >>>>>>>>> reading:
> > > > >>>>>>>>>
> > > > >>>>>>>>> # Read the first record
> > > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > >>>>>>>>> ...
> > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > >>>>>>>>> ...
> > > > >>>>>>>>> # Read the second record
> > > > >>>>>>>>> plot_data_plane
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > >>>>>>>>> ...
> > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > >>>>>>>>>
> > > > >>>>>>>>> Hope that helps.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks,
> > > > >>>>>>>>> John
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran - NOAA
> > > > >>>>>>>>> Affiliate <
> > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi, John,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> When I implemented all the changes (and also set it
up so
> > > > >>>>>>>>>> it's
> > > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > >>>>>>>>>>
> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > >>>>>>>>>> -v 4
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> But then I got this error:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> DEBUG 2:
> > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> > > > >>>>>>>>>> matching
> > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > >>>>>>>>>> WARNING:
> > > > >>>>>>>>>> ERROR :
> > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no requested
> > > > >>>>>>>>>> forecast
> > > > >>>>>>>>>> data
> > > > >>>>>>>>>> found! Exiting...
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2 output
for the
> > > > >>>>>>>>>> file
> > > > >>>>>>>>>> in this case reads:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 14 (t14a2)
> > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12 >
> > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > >>>>>>>>>> 15 (t14a2)
> > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You had
> > > > >>>>>>>>>> suggested
> > > > >>>>>>>>>> that it should be the time stamp of the 2nd day,
but maybe
> > > > >>>>>>>>>> it needs to be
> > > > >>>>>>>>>> the time stamp of the first day?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Unless I'm missing something in pb2nc or elsewhere.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks for any thoughts.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Perry
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley Gotway
<
> > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Perry,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did, we
> > > > >>>>>>>>>>> used
> > > > >>>>>>>>>>> observation files named:
> > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> And those files don't exist in the July data you
just
> > > > >>>>>>>>>>> sent.
> > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
aqm.t12z.anowpm.pb.tm00
> > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 aqm.t12z.anowpm.pb.tm120
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I'm not sure if you're actually verifying against
the
> > > > >>>>>>>>>>> BUFR
> > > > >>>>>>>>>>> or
> > > > >>>>>>>>>>> PREPBUFR.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Let me just tell you the key points for the config
file
> > > > >>>>>>>>>>> settings...
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2
input
> > > > >>>>>>>>>>> files
> > > > >>>>>>>>>>> -
> > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > >>>>>>>>>>> pb2nc \
> > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > >>>>>>>>>>> PB2NCConfig \
> > > > >>>>>>>>>>> -pbfile hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > > >>>>>>>>>>> like
> > > > >>>>>>>>>>> this:
> > > > >>>>>>>>>>> time_summary = {
> > > > >>>>>>>>>>> flag = TRUE;
> > > > >>>>>>>>>>> beg = "040000";
> > > > >>>>>>>>>>> end = "040000";
> > > > >>>>>>>>>>> step = 24*3600;
> > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; } //
> > > > >>>>>>>>>>> previous
> > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > >>>>>>>>>>> grib_code = [];
> > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every
hour
> > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute summaries
when at
> > > > >>>>>>>>>>> least
> > > > >>>>>>>>>>> 75% of expect obs are present
> > > > >>>>>>>>>>> }
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > > > >>>>>>>>>>> over the
> > > > >>>>>>>>>>> defined time period. They're identified in the
output by
> > > > >>>>>>>>>>> the message type:
> > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst"
and "obs"
> > > > >>>>>>>>>>> sections like this for the max of the 1-hour data.
> > > > >>>>>>>>>>> Instead
> > > > >>>>>>>>>>> of specifying
> > > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
convenient
> > > > >>>>>>>>>>> to specify the
> > > > >>>>>>>>>>> valid time. So that you don't have to modify the
config
> > > > >>>>>>>>>>> file, I'd suggest
> > > > >>>>>>>>>>> using environment variables:
> > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> fcst = {
> > > > >>>>>>>>>>> field = [
> > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > >>>>>>>>>>> ];
> > > > >>>>>>>>>>> }
> > > > >>>>>>>>>>> obs = {
> > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > >>>>>>>>>>> field = [
> > > > >>>>>>>>>>> { name = "COPO"; level = "A1"; message_type
=
> > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > >>>>>>>>>>> ];
> > > > >>>>>>>>>>> }
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley Gotway
<
> > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> John
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > > >>>>>>>>>>>> Affiliate <
> > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc
grids and
> > > > >>>>>>>>>>>>> focus on
> > > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files
> > > > >>>>>>>>>>>>> over
> > > > >>>>>>>>>>>>> to Theia so we'll
> > > > >>>>>>>>>>>>> let him work on the files he has for now and if
he runs
> > > > >>>>>>>>>>>>> into problems, I'll
> > > > >>>>>>>>>>>>> provide him with the para5 runs (at least the
24-hr
> > > > >>>>>>>>>>>>> max/avg files).
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not for
> > > > >>>>>>>>>>>>>> bias
> > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe,
> > > > >>>>>>>>>>>>>> you
> > > > >>>>>>>>>>>>>> may want to provide
> > > > >>>>>>>>>>>>>> him para5 runs.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia
for John
> > > > >>>>>>>>>>>>>> to do
> > > > >>>>>>>>>>>>>> some testing some production files. Do you
think I
> > > > >>>>>>>>>>>>>> should use parallels
> > > > >>>>>>>>>>>>>> instead or do you think the ops files are OK
for
> > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > >>>>>>>>>>>>>> for AQ?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the
ops files
> > > > >>>>>>>>>>>>>> should
> > > > >>>>>>>>>>>>>> be OK for now.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen <
> > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels.
> > > > >>>>>>>>>>>>>>> We
> > > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > > > >>>>>>>>>>>>>>> but
> > > > >>>>>>>>>>>>>>> management did not
> > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > > >>>>>>>>>>>>>>> Jeff
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
Affiliate
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
correct time
> > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send
John over
> > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I don't think that the MET team has any other
> > > > >>>>>>>>>>>>>>> versions
> > > > >>>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but
they can
> > > > >>>>>>>>>>>>>>> also clarify this as
> > > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new does
over
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM, jeff.mcqueen
<
> > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may want
to use
> > > > >>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > > > >>>>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management
> > > > >>>>>>>>>>>>>>>> did
> > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > >>>>>>>>>>>>>>>> further delays so we will have to do so with
the
> > > > >>>>>>>>>>>>>>>> FYQ4
> > > > >>>>>>>>>>>>>>>> implementation.
> > > > >>>>>>>>>>>>>>>> Jeff
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
Affiliate
> > > > >>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the conversation.
I
> > > > >>>>>>>>>>>>>>>> think
> > > > >>>>>>>>>>>>>>>> I
> > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some
time ago,
> > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff. I
think
> > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
entirely
> > > > >>>>>>>>>>>>>>>> sure.
> > > > >>>>>>>>>>>>>>>> But he would have
> > > > >>>>>>>>>>>>>>>> more insight as to what is correct and how to
> > > > >>>>>>>>>>>>>>>> approach
> > > > >>>>>>>>>>>>>>>> this.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp issues
John
> > > > >>>>>>>>>>>>>>>> talks
> > > > >>>>>>>>>>>>>>>> about?
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Perry
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
Gotway <
> > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away to
> > > > >>>>>>>>>>>>>>>>> compute
> > > > >>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1-
and 8-
> > > > >>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
\
> > > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > >>>>>>>>>>>>>>>>> -pbfile
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/point_stat
> \
> > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1, for
> > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > > >>>>>>>>>>>>>>>>> FULL,
> > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the contents
of
> > > > >>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>> "max"
> > > > >>>>>>>>>>>>>>>>> file:
> > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave fcst:
> > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
level:17-40
> > > > >>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > > > >>>>>>>>>>>>>>>>> revealed
> > > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours 17
> > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in the
same
> > > > >>>>>>>>>>>>>>>>> file, we
> > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it
which one
> > > > >>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > >>>>>>>>>>>>>>>>> field = [
> > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
lead_time
> > > > >>>>>>>>>>>>>>>>> =
> > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > >>>>>>>>>>>>>>>>> ];
> > > > >>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation timestamps
are off
> > > > >>>>>>>>>>>>>>>>> by 1
> > > > >>>>>>>>>>>>>>>>> hour.
> > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead
> > > > >>>>>>>>>>>>>>>>> time
> > > > >>>>>>>>>>>>>>>>> =
> > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour max
value
> > > > >>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at 05Z
while
> > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-Stat
config
> > > > >>>>>>>>>>>>>>>>> file by
> > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1 hour:
> > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > >>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > >>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries, but I
> > > > >>>>>>>>>>>>>>>>>> found
> > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on
dakota
> > > > >>>>>>>>>>>>>>>>>> here:
> > > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute 24-
> > > > >>>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped the
output
> > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands to
count
> > > > >>>>>>>>>>>>>>>>>> up
> > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > >>>>>>>>>>>>>>>>>> wc
> > > > >>>>>>>>>>>>>>>>>> -l
> > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
pb2nc_output.txt | wc
> > > > >>>>>>>>>>>>>>>>>> -l
> > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > >>>>>>>>>>>>>>>>>> sort
> > > > >>>>>>>>>>>>>>>>>> -u |
> > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538... meaning
that
> > > > >>>>>>>>>>>>>>>>>> all
> > > > >>>>>>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > > > >>>>>>>>>>>>>>>>>> meaning
> > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations are
> > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that
file is
> > > > >>>>>>>>>>>>>>>>>> named
> > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180527_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC 20180528_040000
32.47
> > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated
in the
> > > > >>>>>>>>>>>>>>>>>> last two
> > > > >>>>>>>>>>>>>>>>>> lines.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
duplicates
> > > > >>>>>>>>>>>>>>>>>> are
> > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no reason
why
> > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > >>>>>>>>>>>>>>>>>>> have
> > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard Soh
<
> > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
(550.319
> > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not check
in the
> > > > >>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content of
> > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
inputs. I
> > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files
and will
> > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley Gotway
wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change the
> > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard
Soh <
> > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the
logic for
> > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to compute
24-hour
> > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
counts:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript to
dump
> > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in
PB2NC not
> > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or in
the
> > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard Soh <
> > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> aqm.20180527/aqm.t12z.prepbufr.tm00
and
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM John
Halley
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC. I
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like this:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600;
end = 0;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations...
but
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not what
you'd
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
"AIRNOW_MAX_240000"
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem
caused by
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2 of
them to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM
John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute a
24
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next
week.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM
hsoh <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation
data in
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file was
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
without any
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the
NetCDF
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from
0 to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
summary)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable. The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger
than 3655.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John Halley
Gotway
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota in
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it to
ascii:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
/d1/johnhg/MET/MET_development/
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
pb2nc_output.txt
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists
all of
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But
some of
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as evidenced
by
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
pb2nc_output.nc
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> >
> >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri May 03 10:06:11 2019
Hi, John,
Well, I am running the script in batch so I don't control what node it
gets. I can tell you that the working directory is here:
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
And the output log file is here:
/ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688, which
should
help you figure out how the thing is run.
Thanks!
Perry
On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry,
>
> We can log on to wcoss and test there instead. Yes, pb2nc does seem
to be
> the culprit. Can you point us the wcoss node you're using and the
working
> directory there? We basically need to recreate your call to pb2nc
to try
> to replicate the behavior.
>
> Thanks,
> John
>
> On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >
> > Hi, David, John,
> >
> > I am not running on Theia. In fact I think it would take quite a
bit of
> > effort to transition this thing to Theia, and I'll need to be
running
> this
> > verification on prod data anyway, which is only on WCOSS.
> >
> > However, if you wish to troubleshoot on Theia, you should do so,
and I
> > think you have all you need to be able to do so, right?
> >
> > It would seem that the error on my side is in PB2NC, so I'll
probably
> need
> > to make sure I'm running what you're running.
> >
> > Thanks!
> >
> > Perry
> >
> > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Perry -
> > > Are you running on theia?
> > > John suggests that we try to troubleshoot there ...
> > > thanks,
> > > David
> > >
> > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > Hi Perry -
> > > >
> > > > An update on where we are on this issue:
> > > > Using the prepda.2019042312.nc file you provided we indeed
reproduce
> > > > the point_stat error.
> > > > The point_stat error is due to only a single entry the valid
time
> > > > table,
> > > > e.g. from ncdump -v hdr_vld_table
> > > > hdr_vld_table =
> > > > "20190423_050000",
> > > > while the hdr_vld indices in that file are hdr_vld = 0, 33,
33, ...
> > > > which are all out of range except the first.
> > > >
> > > > If I run pb2nc here with your PREPBUFR data and your
PB2NCConfig_aqm
> > > > config file, however,
> > > > I generate a different prepda.2019042312.nc file that runs
through
> to
> > > > completion with point_stat.
> > > > Its valid time table:
> > > >
> > > > hdr_vld_table =
> > > > "20190423_030000",
> > > > "20190423_010000",
> > > > "20190423_020000",
> > > > "20190423_230000",
> > > > "20190423_150000",
> > > > "20190423_090000",
> > > > "20190423_070000",
> > > > "20190423_060000",
> > > > "20190423_210000",
> > > > "20190423_050000",
> > > > "20190423_170000",
> > > > "20190423_100000",
> > > > "20190423_200000",
> > > > "20190423_140000",
> > > > "20190423_130000",
> > > > "20190423_160000",
> > > > "20190423_040000",
> > > > "20190423_180000",
> > > > "20190423_080000",
> > > > "20190423_120000",
> > > > "20190423_110000",
> > > > "20190423_220000",
> > > > "20190423_190000",
> > > > "20190424_000000" ;
> > > > }
> > > >
> > > > So it seems pb2nc is running correctly here, but not on your
side.
> > > > Let me discuss with John the best way to proceed ...
> > > > thanks,
> > > > David
> > > >
> > > >
> > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > Hi, everyone,
> > > > >
> > > > > I am attaching this to this long thread regarding the daily
max of
> > > > > ozone
> > > > > verification, so we remember what we were doing back then.
> > > > >
> > > > > I am now running this again using MET 8.0, and something
seems to
> not
> > > > > be
> > > > > working when I run point_stat. I get this error when
running a
> daily
> > > > > max
> > > > > of ozone:
> > > > >
> > > > > ERROR :
> > > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > > ERROR :
> > > > >
> > > > > I am using my own scripts to run this using MET, so this is
not
> using
> > > > > METplus.
> > > > >
> > > > > You can find a log file of this here:
> > > > >
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > It
> > > > > runs pb2nc and point_stat for max ozone. The pb2nc code is
working
> > > > > fine, I
> > > > > believe.
> > > > >
> > > > > The script I am using to run this is here:
> > > > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > >
> > > > > The prepbufr files are:
> > > > >
> > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > >
> > > > > The model file is here:
> > > > >
> > > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > >
> > > > >
> > > > > What could be causing this error?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Perry
> > > > >
> > > > >
> > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> johnhg at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Great, glad that did the trick.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
Affiliate <
> > > > > > perry.shafran at noaa.gov> wrote:
> > > > > >
> > > > > >> Well this worked! And Jeff confirmed to me that the 8-hr
> average
> > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in
the 1-hr
> > > > > >> avg.
> > > > > >>
> > > > > >> Thanks!
> > > > > >>
> > > > > >> Perry
> > > > > >>
> > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > >> <johnhg at ucar.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Perry,
> > > > > >>>
> > > > > >>> I think the problem is in your setting of
> > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > >>>
> > > > > >>> Here's your wgrib2 output:
> > > > > >>> wgrib2 oz_DAY1
> > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> > > > > >>> fcst:
> > > > > >>>
> > > > > >>> The "d=2018081112" tells you the model initialization
time.
> The
> > > > > >>> valid
> > > > > >>> time = init time + forecast time. So the OZMAX1 data is
valid
> 16
> > > > > >>> hours
> > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> that
> > > > > >>> wgrib2
> > > > > >>> command using the "-vt" argument to list the
verification time.
> > > > > >>>
> > > > > >>> Point-Stat isn't finding any OZMAX8 data because that
data
> isn't
> > > > > >>> valid
> > > > > >>> at the time you're requesting.
> > > > > >>>
> > > > > >>> John
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
Affiliate
> <
> > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > >>>
> > > > > >>>> Hi, John,
> > > > > >>>>
> > > > > >>>> I haven't been able to work on this for awhile, but now
I'm
> > > > > >>>> working on
> > > > > >>>> it again but seem to be running into issues.
> > > > > >>>>
> > > > > >>>> I have a file that has both the daily max for 1-hr
ozone and
> the
> > > > > >>>> daily
> > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like this:
> > > > > >>>>
> > > > > >>>> 9 (t10a2)
> > > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > >>>> >
> > > > > >>>> wgrib2 oz_DAY1
> > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> fcst:
> > > > > >>>> 10 (t10a2)
> > > > > >>>> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > >>>> >
> > > > > >>>>
> > > > > >>>> As you can see they have different forecast times. But
when I
> > > > > >>>> run
> > > > > >>>> point_stat, I get this output:
> > > > > >>>>
> > > > > >>>> DEBUG 2:
> > > > > >>>> DEBUG 2:
> > > > > >>>>
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > >>>> DEBUG 2:
> > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > >>>> WARNING:
> > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > > > >>>> records
> > > > > >>>> found for 'OZMAX8/L1'
> > > > > >>>> WARNING:
> > > > > >>>> WARNING:
> > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > >>>> OZMAX8/L1
> > > > > >>>> found in file: oz_DAY1
> > > > > >>>> WARNING:
> > > > > >>>>
> > > > > >>>> Here is the relevant portion of the point_stat config
file:
> > > > > >>>>
> > > > > >>>> fcst = {
> > > > > >>>> field = [
> > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > >>>> ];
> > > > > >>>> }
> > > > > >>>>
> > > > > >>>> obs = {
> > > > > >>>> convert(x) = x * 10^9;
> > > > > >>>> field = [
> > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > >>>> ];
> > > > > >>>> }
> > > > > >>>>
> > > > > >>>> I don't remember if I had this working previously for
both the
> > > > > >>>> 1hr
> > > > > >>>> and
> > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue for
> > > > > >>>> the
> > > > > >>>> 8hr
> > > > > >>>> ozone. Have any thoughts?
> > > > > >>>>
> > > > > >>>> Thanks!
> > > > > >>>>
> > > > > >>>> Perry
> > > > > >>>>
> > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
> Affiliate
> > > > > >>>> <
> > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>
> > > > > >>>>> I chose option 2, and that seems to work - I have a
.stat
> file
> > > > > >>>>> with
> > > > > >>>>> output. Now I'll need to do some work here to see if
the
> > > > > >>>>> output
> > > > > >>>>> looks OK,
> > > > > >>>>> and to configure the verification so it verifies
OZMAX1 and
> > > > > >>>>> OZMAX8.
> > > > > >>>>>
> > > > > >>>>> I also wonder if this would be easily modified for PM.
> > > > > >>>>>
> > > > > >>>>> Perry
> > > > > >>>>>
> > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > > > >>>>> <johnhg at ucar.edu>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Perry,
> > > > > >>>>>>
> > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > > > >>>>>> that
> > > > > >>>>>> are
> > > > > >>>>>> valid at 04Z. And you're comparing them to forecasts
that
> are
> > > > > >>>>>> purportedly
> > > > > >>>>>> valid at 05Z. And that's the reason why you have so
many
> obs
> > > > > >>>>>> rejected due
> > > > > >>>>>> to the valid time.
> > > > > >>>>>>
> > > > > >>>>>> It's up to you guys to decide what the "correct" way
of
> > > > > >>>>>> handling
> > > > > >>>>>> this
> > > > > >>>>>> is... but here are a couple of options:
> > > > > >>>>>>
> > > > > >>>>>> (1) You could use the existing setup and address this
logic
> in
> > > > > >>>>>> Point-Stat config file. Change the matching time
obs_window
> > > > > >>>>>> from 0 to:
> > > > > >>>>>>
> > > > > >>>>>> obs_window = {
> > > > > >>>>>> beg = 3600;
> > > > > >>>>>> end = 3600;
> > > > > >>>>>> }
> > > > > >>>>>> This tells Point-Stat to match observations which are
> exactly
> > > > > >>>>>> 1-
> > > > > >>>>>> hour
> > > > > >>>>>> prior to the valid time of the forecast.
> > > > > >>>>>>
> > > > > >>>>>> (2) You could rerun PB2NC to compute a time summary
from 05Z
> > > > > >>>>>> to
> > > > > >>>>>> 05Z
> > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting at
> 0:
> > > > > >>>>>>
> > > > > >>>>>> time_summary = {
> > > > > >>>>>> flag = TRUE;
> > > > > >>>>>> beg = "050000";
> > > > > >>>>>> end = "050000";
> > > > > >>>>>> ...
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> John
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran - NOAA
> > > > > >>>>>> Affiliate
> > > > > >>>>>> <
> > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file, but
> > > > > >>>>>>> it's
> > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > >>>>>>>
> > > > > >>>>>>> DEBUG 4:
> > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> observation
> > > > > >>>>>>> type
> > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> method
> > > > > >>>>>>> NEAREST(1),
> > > > > >>>>>>> using 0 pairs.
> > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > >>>>>>>
> > > > > >>>>>>> Looks like most rejections were due to valid time.
> > > > > >>>>>>>
> > > > > >>>>>>> Have any thoughts about this one?
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks!
> > > > > >>>>>>>
> > > > > >>>>>>> Perry
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > >>>>>>> Affiliate
> > > > > >>>>>>> <
> > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Hi, John,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Looks like the changing of the time stamp from _04
to _05
> > > > > >>>>>>>> did
> > > > > >>>>>>>> the
> > > > > >>>>>>>> trick as point_stat ran without an issue with that
change.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks!
> > > > > >>>>>>>>
> > > > > >>>>>>>> Perry
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Perry,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I agree, there's likely some issue with requesting
the
> > > > > >>>>>>>>> timestamp
> > > > > >>>>>>>>> of the data from that file. I think switching the
> > > > > >>>>>>>>> verification time to
> > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> initialization
> > > > > >>>>>>>>> times
> > > > > >>>>>>>>> and valid times for this data. Here's some useful
wgrib2
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>> MET commands
> > > > > >>>>>>>>> to illustrate.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > > > >>>>>>>>> time
> > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is using
a
> sample
> > > > > >>>>>>>>> data file you
> > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> wgrib2
> > > > > >>>>>>>>> so
> > > > > >>>>>>>>> I have that
> > > > > >>>>>>>>> funky forecast hour:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour ave
> > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
verbosity
> > > > > >>>>>>>>> level
> > > > > >>>>>>>>> (-v
> > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data it is
> > > > > >>>>>>>>> reading:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> # Read the first record
> > > > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > >>>>>>>>> ...
> > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > >>>>>>>>> ...
> > > > > >>>>>>>>> # Read the second record
> > > > > >>>>>>>>> plot_data_plane
> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > >>>>>>>>> ...
> > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Hope that helps.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks,
> > > > > >>>>>>>>> John
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran -
NOAA
> > > > > >>>>>>>>> Affiliate <
> > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Hi, John,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> When I implemented all the changes (and also set
it up
> so
> > > > > >>>>>>>>>> it's
> > > > > >>>>>>>>>> reading the beta6 version), I used this command:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > >>>>>>>>>>
> > >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > >>>>>>>>>> -v 4
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> But then I got this error:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> DEBUG 2:
> > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
> > > > > >>>>>>>>>> matching
> > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no fields
> matching
> > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > >>>>>>>>>> WARNING:
> > > > > >>>>>>>>>> ERROR :
> > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > >>>>>>>>>> forecast
> > > > > >>>>>>>>>> data
> > > > > >>>>>>>>>> found! Exiting...
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output for
> the
> > > > > >>>>>>>>>> file
> > > > > >>>>>>>>>> in this case reads:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 14 (t14a2)
> > > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > >>>>>>>>>> 15 (t14a2)
> > > > > >>>>>>>>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You
had
> > > > > >>>>>>>>>> suggested
> > > > > >>>>>>>>>> that it should be the time stamp of the 2nd day,
but
> maybe
> > > > > >>>>>>>>>> it needs to be
> > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Perry
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Perry,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I'm confused. In the previous air quality vx we
did,
> we
> > > > > >>>>>>>>>>> used
> > > > > >>>>>>>>>>> observation files named:
> > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> And those files don't exist in the July data you
just
> > > > > >>>>>>>>>>> sent.
> > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
aqm.t12z.anowpm.pb.tm00
> > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against the
> > > > > >>>>>>>>>>> BUFR
> > > > > >>>>>>>>>>> or
> > > > > >>>>>>>>>>> PREPBUFR.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Let me just tell you the key points for the
config file
> > > > > >>>>>>>>>>> settings...
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2
input
> > > > > >>>>>>>>>>> files
> > > > > >>>>>>>>>>> -
> > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > >>>>>>>>>>> pb2nc \
> > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > > > >>>>>>>>>>> like
> > > > > >>>>>>>>>>> this:
> > > > > >>>>>>>>>>> time_summary = {
> > > > > >>>>>>>>>>> flag = TRUE;
> > > > > >>>>>>>>>>> beg = "040000";
> > > > > >>>>>>>>>>> end = "040000";
> > > > > >>>>>>>>>>> step = 24*3600;
> > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; }
//
> > > > > >>>>>>>>>>> previous
> > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > >>>>>>>>>>> grib_code = [];
> > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every
hour
> > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries when
> at
> > > > > >>>>>>>>>>> least
> > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > >>>>>>>>>>> }
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at each
station
> > > > > >>>>>>>>>>> over the
> > > > > >>>>>>>>>>> defined time period. They're identified in the
output
> by
> > > > > >>>>>>>>>>> the message type:
> > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the "fcst"
and
> "obs"
> > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > >>>>>>>>>>> Instead
> > > > > >>>>>>>>>>> of specifying
> > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
> convenient
> > > > > >>>>>>>>>>> to specify the
> > > > > >>>>>>>>>>> valid time. So that you don't have to modify
the
> config
> > > > > >>>>>>>>>>> file, I'd suggest
> > > > > >>>>>>>>>>> using environment variables:
> > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> fcst = {
> > > > > >>>>>>>>>>> field = [
> > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > >>>>>>>>>>> ];
> > > > > >>>>>>>>>>> }
> > > > > >>>>>>>>>>> obs = {
> > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > >>>>>>>>>>> field = [
> > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > >>>>>>>>>>> ];
> > > > > >>>>>>>>>>> }
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway <
> > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran -
NOAA
> > > > > >>>>>>>>>>>> Affiliate <
> > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc
grids and
> > > > > >>>>>>>>>>>>> focus on
> > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to copy
files
> > > > > >>>>>>>>>>>>> over
> > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > >>>>>>>>>>>>> let him work on the files he has for now and
if he
> runs
> > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least the
24-hr
> > > > > >>>>>>>>>>>>> max/avg files).
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks!
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen <
> > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not for
> > > > > >>>>>>>>>>>>>> bias
> > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To be
safe,
> > > > > >>>>>>>>>>>>>> you
> > > > > >>>>>>>>>>>>>> may want to provide
> > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
Affiliate
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to Theia
for
> John
> > > > > >>>>>>>>>>>>>> to do
> > > > > >>>>>>>>>>>>>> some testing some production files. Do you
think I
> > > > > >>>>>>>>>>>>>> should use parallels
> > > > > >>>>>>>>>>>>>> instead or do you think the ops files are OK
for
> > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > >>>>>>>>>>>>>> for AQ?
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that the
ops
> files
> > > > > >>>>>>>>>>>>>> should
> > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM, jeff.mcqueen
<
> > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
parallels.
> > > > > >>>>>>>>>>>>>>> We
> > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max hour
labels
> > > > > >>>>>>>>>>>>>>> but
> > > > > >>>>>>>>>>>>>>> management did not
> > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > > > >>>>>>>>>>>>>>> Jeff
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
> Affiliate
> > > > > >>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
correct
> time
> > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send
John
> over
> > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > >>>>>>>>>>>>>>> versions
> > > > > >>>>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have - but
they
> can
> > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new
does over
> > > > > >>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want to
> use
> > > > > >>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
Jianping
> > > > > >>>>>>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
management
> > > > > >>>>>>>>>>>>>>>> did
> > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with the
> > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
> Affiliate
> > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation. I
> > > > > >>>>>>>>>>>>>>>> think
> > > > > >>>>>>>>>>>>>>>> I
> > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps some
time
> ago,
> > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with Jeff.
I
> think
> > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
entirely
> > > > > >>>>>>>>>>>>>>>> sure.
> > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and how
to
> > > > > >>>>>>>>>>>>>>>> approach
> > > > > >>>>>>>>>>>>>>>> this.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues John
> > > > > >>>>>>>>>>>>>>>> talks
> > > > > >>>>>>>>>>>>>>>> about?
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Perry
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John Halley
> Gotway <
> > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared away
to
> > > > > >>>>>>>>>>>>>>>>> compute
> > > > > >>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for 1-
and 8-
> > > > > >>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > > > >>>>>>>>>>>>>>>>> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > >>>>>>>>>>>>>>>>> -pbfile
> aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > \
> > > > > >>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1, for
> > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents of
> > > > > >>>>>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > >>>>>>>>>>>>>>>>> file:
> > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> level:17-40
> > > > > >>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
testing
> > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being forecast
hours
> 17
> > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
better.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in
the same
> > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it
which one
> > > > > >>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> lead_time
> > > > > >>>>>>>>>>>>>>>>> =
> > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > >>>>>>>>>>>>>>>>> ];
> > > > > >>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps are
> off
> > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41 hour
lead
> > > > > >>>>>>>>>>>>>>>>> time
> > > > > >>>>>>>>>>>>>>>>> =
> > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour
max
> value
> > > > > >>>>>>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at
05Z
> while
> > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-
Stat
> config
> > > > > >>>>>>>>>>>>>>>>> file by
> > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > >>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and was
fixed.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley Gotway
wrote:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> but I
> > > > > >>>>>>>>>>>>>>>>>> found
> > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on
dakota
> > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > >>>>>>>>>>>>>>>>>>
/d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute 24-
> > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped
the
> output
> > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands
to count
> > > > > >>>>>>>>>>>>>>>>>> up
> > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt
> |
> > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
pb2nc_output.txt
> |
> > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> wc
> > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning that
> > > > > >>>>>>>>>>>>>>>>>> all
> > > > > >>>>>>>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4) returns
6,555
> > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations
are
> > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in that
file is
> > > > > >>>>>>>>>>>>>>>>>> named
> > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt |
grep
> > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000 32.47
> > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are repeated
in the
> > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
> duplicates
> > > > > >>>>>>>>>>>>>>>>>> are
> > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley Gotway
wrote:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason why
> > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
across 2
> > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which we
could
> > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard
Soh <
> > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10 minutes
> (550.319
> > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check in
> the
> > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content
> of
> > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
summary
> > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
inputs. I
> > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input files
and
> will
> > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change the
> > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
currently
> > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM Howard
Soh <
> > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
summary
> > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM John
Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the
logic
> for
> > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> 24-hour
> > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these obs/hdr
> counts:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the Rscript
to dump
> > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the AIRNOW_MAX_240000
string
> > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in
PB2NC not
> > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or
in the
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look in
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM John
Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> Soh <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations from
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00 and
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the obs
data.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM
John
> Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could you
please
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC.
> I
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request 24-
hourly
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg = -24*3600;
end =
> 0;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295 observations...
but
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what you'd
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> "AIRNOW_MAX_240000"
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you pointed
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem
caused by
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2
of them
> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM
John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
PrepBUFR
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to compute
a 24
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next
week.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40 PM
hsoh <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
reasonable
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no observation
data in
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file
was
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
without
> any
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
headers
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS. 118193=headers).
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the
NetCDF
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
currently)
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value from
0 to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type for
> summary)
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable. The
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger
than
> 3655.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be OK.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> Gotway
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota
> in
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump it
to
> ascii:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> /d1/johnhg/MET/MET_development/
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
pb2nc_output.txt
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but lists
all of
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But
some of
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced by
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> pb2nc_output.nc
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing or
table
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>
> > >
> > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Mon Jun 03 12:16:16 2019
Perry,
Did you ever get this issue resolved from 1 month ago? Or should I
log on
to WCOSS and take a look?
Thanks,
John
On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
> Hi, John,
>
> Well, I am running the script in batch so I don't control what node
it
> gets. I can tell you that the working directory is here:
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>
> And the output log file is here:
> /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688, which
should
> help you figure out how the thing is run.
>
> Thanks!
>
> Perry
>
> On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Perry,
> >
> > We can log on to wcoss and test there instead. Yes, pb2nc does
seem to
> be
> > the culprit. Can you point us the wcoss node you're using and the
> working
> > directory there? We basically need to recreate your call to pb2nc
to try
> > to replicate the behavior.
> >
> > Thanks,
> > John
> >
> > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > >
> > > Hi, David, John,
> > >
> > > I am not running on Theia. In fact I think it would take quite
a bit
> of
> > > effort to transition this thing to Theia, and I'll need to be
running
> > this
> > > verification on prod data anyway, which is only on WCOSS.
> > >
> > > However, if you wish to troubleshoot on Theia, you should do so,
and I
> > > think you have all you need to be able to do so, right?
> > >
> > > It would seem that the error on my side is in PB2NC, so I'll
probably
> > need
> > > to make sure I'm running what you're running.
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Perry -
> > > > Are you running on theia?
> > > > John suggests that we try to troubleshoot there ...
> > > > thanks,
> > > > David
> > > >
> > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > Hi Perry -
> > > > >
> > > > > An update on where we are on this issue:
> > > > > Using the prepda.2019042312.nc file you provided we indeed
> reproduce
> > > > > the point_stat error.
> > > > > The point_stat error is due to only a single entry the valid
time
> > > > > table,
> > > > > e.g. from ncdump -v hdr_vld_table
> > > > > hdr_vld_table =
> > > > > "20190423_050000",
> > > > > while the hdr_vld indices in that file are hdr_vld = 0, 33,
33, ...
> > > > > which are all out of range except the first.
> > > > >
> > > > > If I run pb2nc here with your PREPBUFR data and your
> PB2NCConfig_aqm
> > > > > config file, however,
> > > > > I generate a different prepda.2019042312.nc file that runs
through
> > to
> > > > > completion with point_stat.
> > > > > Its valid time table:
> > > > >
> > > > > hdr_vld_table =
> > > > > "20190423_030000",
> > > > > "20190423_010000",
> > > > > "20190423_020000",
> > > > > "20190423_230000",
> > > > > "20190423_150000",
> > > > > "20190423_090000",
> > > > > "20190423_070000",
> > > > > "20190423_060000",
> > > > > "20190423_210000",
> > > > > "20190423_050000",
> > > > > "20190423_170000",
> > > > > "20190423_100000",
> > > > > "20190423_200000",
> > > > > "20190423_140000",
> > > > > "20190423_130000",
> > > > > "20190423_160000",
> > > > > "20190423_040000",
> > > > > "20190423_180000",
> > > > > "20190423_080000",
> > > > > "20190423_120000",
> > > > > "20190423_110000",
> > > > > "20190423_220000",
> > > > > "20190423_190000",
> > > > > "20190424_000000" ;
> > > > > }
> > > > >
> > > > > So it seems pb2nc is running correctly here, but not on your
side.
> > > > > Let me discuss with John the best way to proceed ...
> > > > > thanks,
> > > > > David
> > > > >
> > > > >
> > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > > Hi, everyone,
> > > > > >
> > > > > > I am attaching this to this long thread regarding the
daily max
> of
> > > > > > ozone
> > > > > > verification, so we remember what we were doing back then.
> > > > > >
> > > > > > I am now running this again using MET 8.0, and something
seems to
> > not
> > > > > > be
> > > > > > working when I run point_stat. I get this error when
running a
> > daily
> > > > > > max
> > > > > > of ozone:
> > > > > >
> > > > > > ERROR :
> > > > > > ERROR : StringArray::operator[](int) const -> range check
error!
> > > > > > ERROR :
> > > > > >
> > > > > > I am using my own scripts to run this using MET, so this
is not
> > using
> > > > > > METplus.
> > > > > >
> > > > > > You can find a log file of this here:
> > > > > >
> > >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > It
> > > > > > runs pb2nc and point_stat for max ozone. The pb2nc code is
> working
> > > > > > fine, I
> > > > > > believe.
> > > > > >
> > > > > > The script I am using to run this is here:
> > > > > >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > >
> > > > > > The prepbufr files are:
> > > > > >
> > > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
and
> > > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > >
> > > > > > The model file is here:
> > > > > >
> > > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > >
> > > > > >
> > > > > > What could be causing this error?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Perry
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > johnhg at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Great, glad that did the trick.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
Affiliate
> <
> > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > >
> > > > > > >> Well this worked! And Jeff confirmed to me that the 8-
hr
> > average
> > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like in
the
> 1-hr
> > > > > > >> avg.
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >> Perry
> > > > > > >>
> > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > >> <johnhg at ucar.edu>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Perry,
> > > > > > >>>
> > > > > > >>> I think the problem is in your setting of
> > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > >>>
> > > > > > >>> Here's your wgrib2 output:
> > > > > > >>> wgrib2 oz_DAY1
> > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour ave
> > > > > > >>> fcst:
> > > > > > >>>
> > > > > > >>> The "d=2018081112" tells you the model initialization
time.
> > The
> > > > > > >>> valid
> > > > > > >>> time = init time + forecast time. So the OZMAX1 data
is
> valid
> > 16
> > > > > > >>> hours
> > > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> > that
> > > > > > >>> wgrib2
> > > > > > >>> command using the "-vt" argument to list the
verification
> time.
> > > > > > >>>
> > > > > > >>> Point-Stat isn't finding any OZMAX8 data because that
data
> > isn't
> > > > > > >>> valid
> > > > > > >>> at the time you're requesting.
> > > > > > >>>
> > > > > > >>> John
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
> Affiliate
> > <
> > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>
> > > > > > >>>> Hi, John,
> > > > > > >>>>
> > > > > > >>>> I haven't been able to work on this for awhile, but
now I'm
> > > > > > >>>> working on
> > > > > > >>>> it again but seem to be running into issues.
> > > > > > >>>>
> > > > > > >>>> I have a file that has both the daily max for 1-hr
ozone and
> > the
> > > > > > >>>> daily
> > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > >>>>
> > > > > > >>>> 9 (t10a2)
> > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > >>>> >
> > > > > > >>>> wgrib2 oz_DAY1
> > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour ave
fcst:
> > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22 hour
ave
> > fcst:
> > > > > > >>>> 10 (t10a2)
> > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > >>>> >
> > > > > > >>>>
> > > > > > >>>> As you can see they have different forecast times.
But
> when I
> > > > > > >>>> run
> > > > > > >>>> point_stat, I get this output:
> > > > > > >>>>
> > > > > > >>>> DEBUG 2:
> > > > > > >>>> DEBUG 2:
> > > > > > >>>>
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > >>>> DEBUG 2:
> > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > >>>> WARNING:
> > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
matching
> > > > > > >>>> records
> > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > >>>> WARNING:
> > > > > > >>>> WARNING:
> > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > > >>>> OZMAX8/L1
> > > > > > >>>> found in file: oz_DAY1
> > > > > > >>>> WARNING:
> > > > > > >>>>
> > > > > > >>>> Here is the relevant portion of the point_stat config
file:
> > > > > > >>>>
> > > > > > >>>> fcst = {
> > > > > > >>>> field = [
> > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > >>>> ];
> > > > > > >>>> }
> > > > > > >>>>
> > > > > > >>>> obs = {
> > > > > > >>>> convert(x) = x * 10^9;
> > > > > > >>>> field = [
> > > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > >>>> ];
> > > > > > >>>> }
> > > > > > >>>>
> > > > > > >>>> I don't remember if I had this working previously for
both
> the
> > > > > > >>>> 1hr
> > > > > > >>>> and
> > > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue
> for
> > > > > > >>>> the
> > > > > > >>>> 8hr
> > > > > > >>>> ozone. Have any thoughts?
> > > > > > >>>>
> > > > > > >>>> Thanks!
> > > > > > >>>>
> > > > > > >>>> Perry
> > > > > > >>>>
> > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > >>>> <
> > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>
> > > > > > >>>>> I chose option 2, and that seems to work - I have a
.stat
> > file
> > > > > > >>>>> with
> > > > > > >>>>> output. Now I'll need to do some work here to see
if the
> > > > > > >>>>> output
> > > > > > >>>>> looks OK,
> > > > > > >>>>> and to configure the verification so it verifies
OZMAX1 and
> > > > > > >>>>> OZMAX8.
> > > > > > >>>>>
> > > > > > >>>>> I also wonder if this would be easily modified for
PM.
> > > > > > >>>>>
> > > > > > >>>>> Perry
> > > > > > >>>>>
> > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley Gotway
> > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Perry,
> > > > > > >>>>>>
> > > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
summaries
> > > > > > >>>>>> that
> > > > > > >>>>>> are
> > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts that
> > are
> > > > > > >>>>>> purportedly
> > > > > > >>>>>> valid at 05Z. And that's the reason why you have
so many
> > obs
> > > > > > >>>>>> rejected due
> > > > > > >>>>>> to the valid time.
> > > > > > >>>>>>
> > > > > > >>>>>> It's up to you guys to decide what the "correct"
way of
> > > > > > >>>>>> handling
> > > > > > >>>>>> this
> > > > > > >>>>>> is... but here are a couple of options:
> > > > > > >>>>>>
> > > > > > >>>>>> (1) You could use the existing setup and address
this
> logic
> > in
> > > > > > >>>>>> Point-Stat config file. Change the matching time
> obs_window
> > > > > > >>>>>> from 0 to:
> > > > > > >>>>>>
> > > > > > >>>>>> obs_window = {
> > > > > > >>>>>> beg = 3600;
> > > > > > >>>>>> end = 3600;
> > > > > > >>>>>> }
> > > > > > >>>>>> This tells Point-Stat to match observations which
are
> > exactly
> > > > > > >>>>>> 1-
> > > > > > >>>>>> hour
> > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > >>>>>>
> > > > > > >>>>>> (2) You could rerun PB2NC to compute a time summary
from
> 05Z
> > > > > > >>>>>> to
> > > > > > >>>>>> 05Z
> > > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting
> at
> > 0:
> > > > > > >>>>>>
> > > > > > >>>>>> time_summary = {
> > > > > > >>>>>> flag = TRUE;
> > > > > > >>>>>> beg = "050000";
> > > > > > >>>>>> end = "050000";
> > > > > > >>>>>> ...
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks,
> > > > > > >>>>>> John
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran -
NOAA
> > > > > > >>>>>> Affiliate
> > > > > > >>>>>> <
> > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file,
> but
> > > > > > >>>>>>> it's
> > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > >>>>>>>
> > > > > > >>>>>>> DEBUG 4:
> > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1, for
> > observation
> > > > > > >>>>>>> type
> > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> > method
> > > > > > >>>>>>> NEAREST(1),
> > > > > > >>>>>>> using 0 pairs.
> > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > >>>>>>>
> > > > > > >>>>>>> Looks like most rejections were due to valid time.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks!
> > > > > > >>>>>>>
> > > > > > >>>>>>> Perry
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > > >>>>>>> Affiliate
> > > > > > >>>>>>> <
> > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Hi, John,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Looks like the changing of the time stamp from
_04 to
> _05
> > > > > > >>>>>>>> did
> > > > > > >>>>>>>> the
> > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> change.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks!
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Perry
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Perry,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting the
> > > > > > >>>>>>>>> timestamp
> > > > > > >>>>>>>>> of the data from that file. I think switching
the
> > > > > > >>>>>>>>> verification time to
> > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > initialization
> > > > > > >>>>>>>>> times
> > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> wgrib2
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>> MET commands
> > > > > > >>>>>>>>> to illustrate.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
forecast
> > > > > > >>>>>>>>> time
> > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > sample
> > > > > > >>>>>>>>> data file you
> > > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> > wgrib2
> > > > > > >>>>>>>>> so
> > > > > > >>>>>>>>> I have that
> > > > > > >>>>>>>>> funky forecast hour:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour
ave
> > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
fcst:vt=2018070605
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
verbosity
> > > > > > >>>>>>>>> level
> > > > > > >>>>>>>>> (-v
> > > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data it
> is
> > > > > > >>>>>>>>> reading:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> # Read the first record
> > > > > > >>>>>>>>> plot_data_plane
> > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > >>>>>>>>> ...
> > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > >>>>>>>>> ...
> > > > > > >>>>>>>>> # Read the second record
> > > > > > >>>>>>>>> plot_data_plane
> > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > >>>>>>>>> ...
> > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Hope that helps.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks,
> > > > > > >>>>>>>>> John
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran -
NOAA
> > > > > > >>>>>>>>> Affiliate <
> > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Hi, John,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> When I implemented all the changes (and also
set it up
> > so
> > > > > > >>>>>>>>>> it's
> > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > >>>>>>>>>>
> > > >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > >>>>>>>>>> -v 4
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> But then I got this error:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
> > > > > > >>>>>>>>>> matching
> > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > matching
> > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > >>>>>>>>>> WARNING:
> > > > > > >>>>>>>>>> ERROR :
> > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > > >>>>>>>>>> forecast
> > > > > > >>>>>>>>>> data
> > > > > > >>>>>>>>>> found! Exiting...
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output for
> > the
> > > > > > >>>>>>>>>> file
> > > > > > >>>>>>>>>> in this case reads:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > >>>>>>>>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >
> > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour ave
> > fcst:
> > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > >>>>>>>>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04? You
had
> > > > > > >>>>>>>>>> suggested
> > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day, but
> > maybe
> > > > > > >>>>>>>>>> it needs to be
> > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Perry
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Perry,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I'm confused. In the previous air quality vx
we did,
> > we
> > > > > > >>>>>>>>>>> used
> > > > > > >>>>>>>>>>> observation files named:
> > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> And those files don't exist in the July data
you just
> > > > > > >>>>>>>>>>> sent.
> > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
aqm.t12z.anowpm.pb.tm00
> > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against the
> > > > > > >>>>>>>>>>> BUFR
> > > > > > >>>>>>>>>>> or
> > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Let me just tell you the key points for the
config
> file
> > > > > > >>>>>>>>>>> settings...
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass 2
input
> > > > > > >>>>>>>>>>> files
> > > > > > >>>>>>>>>>> -
> > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > >>>>>>>>>>> pb2nc \
> > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
time_summary
> > > > > > >>>>>>>>>>> like
> > > > > > >>>>>>>>>>> this:
> > > > > > >>>>>>>>>>> time_summary = {
> > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > >>>>>>>>>>> end = "040000";
> > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0; }
//
> > > > > > >>>>>>>>>>> previous
> > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs every
hour
> > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> when
> > at
> > > > > > >>>>>>>>>>> least
> > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > >>>>>>>>>>> }
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> station
> > > > > > >>>>>>>>>>> over the
> > > > > > >>>>>>>>>>> defined time period. They're identified in
the
> output
> > by
> > > > > > >>>>>>>>>>> the message type:
> > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst" and
> > "obs"
> > > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > > >>>>>>>>>>> Instead
> > > > > > >>>>>>>>>>> of specifying
> > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be more
> > convenient
> > > > > > >>>>>>>>>>> to specify the
> > > > > > >>>>>>>>>>> valid time. So that you don't have to modify
the
> > config
> > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > >>>>>>>>>>> using environment variables:
> > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> fcst = {
> > > > > > >>>>>>>>>>> field = [
> > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > >>>>>>>>>>> ];
> > > > > > >>>>>>>>>>> }
> > > > > > >>>>>>>>>>> obs = {
> > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > >>>>>>>>>>> field = [
> > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > >>>>>>>>>>> ];
> > > > > > >>>>>>>>>>> }
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway <
> > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry Shafran
- NOAA
> > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the bc
grids
> and
> > > > > > >>>>>>>>>>>>> focus on
> > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy files
> > > > > > >>>>>>>>>>>>> over
> > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > >>>>>>>>>>>>> let him work on the files he has for now and
if he
> > runs
> > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least
the 24-hr
> > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM, jeff.mcqueen
<
> > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not
> for
> > > > > > >>>>>>>>>>>>>> bias
> > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids, To
be
> safe,
> > > > > > >>>>>>>>>>>>>> you
> > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
> Affiliate
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia for
> > John
> > > > > > >>>>>>>>>>>>>> to do
> > > > > > >>>>>>>>>>>>>> some testing some production files. Do you
think
> I
> > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > >>>>>>>>>>>>>> instead or do you think the ops files are
OK for
> > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that
the ops
> > files
> > > > > > >>>>>>>>>>>>>> should
> > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
> parallels.
> > > > > > >>>>>>>>>>>>>>> We
> > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max
hour
> labels
> > > > > > >>>>>>>>>>>>>>> but
> > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
upgrade.
> > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
correct
> > time
> > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I send
John
> > over
> > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > > >>>>>>>>>>>>>>> versions
> > > > > > >>>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have -
but they
> > can
> > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the wgrib2new
does
> over
> > > > > > >>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want to
> > use
> > > > > > >>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I recall
> Jianping
> > > > > > >>>>>>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
> management
> > > > > > >>>>>>>>>>>>>>>> did
> > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with the
> > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran - NOAA
> > Affiliate
> > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation. I
> > > > > > >>>>>>>>>>>>>>>> think
> > > > > > >>>>>>>>>>>>>>>> I
> > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some time
> > ago,
> > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff. I
> > think
> > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
entirely
> > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and
how to
> > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> John
> > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > Gotway <
> > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away to
> > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for
1- and
> 8-
> > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc \
> > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>>
> > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > \
> > > > > > >>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW \
> > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1,
> for
> > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000, over
region
> > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents of
> > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > >>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> > level:17-40
> > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny. But
> testing
> > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> hours
> > 17
> > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be encoded
> better.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are in
the
> same
> > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell it
which
> one
> > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > lead_time
> > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps are
> > off
> > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41
hour lead
> > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-hour
max
> > value
> > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid at
05Z
> > while
> > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the Point-
Stat
> > config
> > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and
was
> fixed.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> > but I
> > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look on
dakota
> > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > >>>>>>>>>>>>>>>>>>
> /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute
> 24-
> > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I dumped
the
> > output
> > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these commands
to
> count
> > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> pb2nc_output.txt
> > |
> > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> pb2nc_output.txt
> > |
> > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > wc
> > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
pb2nc_output.txt |
> > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> that
> > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> 6,555
> > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000 observations
are
> > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that file
> is
> > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000 pb2nc_output.txt
| grep
> > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> 32.47
> > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated in
> the
> > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why these
> > duplicates
> > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason why
> > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD values
> across 2
> > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which
we could
> > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM Howard
Soh <
> > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > (550.319
> > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check in
> > the
> > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM hsoh
> > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output (the
content
> > of
> > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for time
> summary
> > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
inputs.
> I
> > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files and
> > will
> > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> > wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change
> the
> > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
> currently
> > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard Soh
> <
> > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for time
> summary
> > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to the
logic
> > for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > 24-hour
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > counts:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
currently)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript to
> dump
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> string
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is in
PB2NC
> not
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries... or
in the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another look
in
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM
John
> Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into this.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> > Soh <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> and
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the
obs
> data.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM
John
> > Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could
you
> please
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of running
PB2NC.
> > I
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request
> 24-hourly
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600; end =
> > 0;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations... but
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what
> you'd
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > "AIRNOW_MAX_240000"
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a problem
caused
> by
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need 2
of
> them
> > to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44 PM
John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an AIRNOW
> PrepBUFR
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute a 24
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together next
week.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40
PM hsoh
> <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
> reasonable
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation data
> in
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index of
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF file
was
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
without
> > any
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has MORE
> headers
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from the
NetCDF
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> currently)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value
from 0
> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type
for
> > summary)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable.
> The
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is bigger
than
> > 3655.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be
OK.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> > Gotway
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look on
dakota
> > in
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump
it to
> > ascii:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > /d1/johnhg/MET/MET_development/
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
pb2nc_output.txt
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists all
> of
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW. But
some
> of
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced by
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > pb2nc_output.nc
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing
or table
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Mon Jun 03 12:57:27 2019
Hi, John,
To be honest, I haven't gotten a chance to look at it. Now that MET
8.1 is
up, I will try it again in the next day or two and let you know.
Thanks for reminding me!
Perry
On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry,
>
> Did you ever get this issue resolved from 1 month ago? Or should I
log on
> to WCOSS and take a look?
>
> Thanks,
> John
>
> On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> > Hi, John,
> >
> > Well, I am running the script in batch so I don't control what
node it
> > gets. I can tell you that the working directory is here:
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > And the output log file is here:
> > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
> should
> > help you figure out how the thing is run.
> >
> > Thanks!
> >
> > Perry
> >
> > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Perry,
> > >
> > > We can log on to wcoss and test there instead. Yes, pb2nc does
seem to
> > be
> > > the culprit. Can you point us the wcoss node you're using and
the
> > working
> > > directory there? We basically need to recreate your call to
pb2nc to
> try
> > > to replicate the behavior.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > Hi, David, John,
> > > >
> > > > I am not running on Theia. In fact I think it would take
quite a bit
> > of
> > > > effort to transition this thing to Theia, and I'll need to be
running
> > > this
> > > > verification on prod data anyway, which is only on WCOSS.
> > > >
> > > > However, if you wish to troubleshoot on Theia, you should do
so, and
> I
> > > > think you have all you need to be able to do so, right?
> > > >
> > > > It would seem that the error on my side is in PB2NC, so I'll
probably
> > > need
> > > > to make sure I'm running what you're running.
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Perry -
> > > > > Are you running on theia?
> > > > > John suggests that we try to troubleshoot there ...
> > > > > thanks,
> > > > > David
> > > > >
> > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > > Hi Perry -
> > > > > >
> > > > > > An update on where we are on this issue:
> > > > > > Using the prepda.2019042312.nc file you provided we indeed
> > reproduce
> > > > > > the point_stat error.
> > > > > > The point_stat error is due to only a single entry the
valid time
> > > > > > table,
> > > > > > e.g. from ncdump -v hdr_vld_table
> > > > > > hdr_vld_table =
> > > > > > "20190423_050000",
> > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33, 33,
> ...
> > > > > > which are all out of range except the first.
> > > > > >
> > > > > > If I run pb2nc here with your PREPBUFR data and your
> > PB2NCConfig_aqm
> > > > > > config file, however,
> > > > > > I generate a different prepda.2019042312.nc file that runs
> through
> > > to
> > > > > > completion with point_stat.
> > > > > > Its valid time table:
> > > > > >
> > > > > > hdr_vld_table =
> > > > > > "20190423_030000",
> > > > > > "20190423_010000",
> > > > > > "20190423_020000",
> > > > > > "20190423_230000",
> > > > > > "20190423_150000",
> > > > > > "20190423_090000",
> > > > > > "20190423_070000",
> > > > > > "20190423_060000",
> > > > > > "20190423_210000",
> > > > > > "20190423_050000",
> > > > > > "20190423_170000",
> > > > > > "20190423_100000",
> > > > > > "20190423_200000",
> > > > > > "20190423_140000",
> > > > > > "20190423_130000",
> > > > > > "20190423_160000",
> > > > > > "20190423_040000",
> > > > > > "20190423_180000",
> > > > > > "20190423_080000",
> > > > > > "20190423_120000",
> > > > > > "20190423_110000",
> > > > > > "20190423_220000",
> > > > > > "20190423_190000",
> > > > > > "20190424_000000" ;
> > > > > > }
> > > > > >
> > > > > > So it seems pb2nc is running correctly here, but not on
your
> side.
> > > > > > Let me discuss with John the best way to proceed ...
> > > > > > thanks,
> > > > > > David
> > > > > >
> > > > > >
> > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > > > Hi, everyone,
> > > > > > >
> > > > > > > I am attaching this to this long thread regarding the
daily max
> > of
> > > > > > > ozone
> > > > > > > verification, so we remember what we were doing back
then.
> > > > > > >
> > > > > > > I am now running this again using MET 8.0, and something
seems
> to
> > > not
> > > > > > > be
> > > > > > > working when I run point_stat. I get this error when
running a
> > > daily
> > > > > > > max
> > > > > > > of ozone:
> > > > > > >
> > > > > > > ERROR :
> > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> error!
> > > > > > > ERROR :
> > > > > > >
> > > > > > > I am using my own scripts to run this using MET, so this
is not
> > > using
> > > > > > > METplus.
> > > > > > >
> > > > > > > You can find a log file of this here:
> > > > > > >
> > > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > > It
> > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc code
is
> > working
> > > > > > > fine, I
> > > > > > > believe.
> > > > > > >
> > > > > > > The script I am using to run this is here:
> > > > > > >
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > > >
> > > > > > > The prepbufr files are:
> > > > > > >
> > > > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
and
> > > > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > > >
> > > > > > > The model file is here:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > > >
> > > > > > >
> > > > > > > What could be causing this error?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Perry
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > > johnhg at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Great, glad that did the trick.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
> Affiliate
> > <
> > > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > > >
> > > > > > > >> Well this worked! And Jeff confirmed to me that the
8-hr
> > > average
> > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like
in the
> > 1-hr
> > > > > > > >> avg.
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> Perry
> > > > > > > >>
> > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > > >> <johnhg at ucar.edu>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Perry,
> > > > > > > >>>
> > > > > > > >>> I think the problem is in your setting of
> > > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > > >>>
> > > > > > > >>> Here's your wgrib2 output:
> > > > > > > >>> wgrib2 oz_DAY1
> > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
> ave
> > > > > > > >>> fcst:
> > > > > > > >>>
> > > > > > > >>> The "d=2018081112" tells you the model
initialization time.
> > > The
> > > > > > > >>> valid
> > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
> > valid
> > > 16
> > > > > > > >>> hours
> > > > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> > > that
> > > > > > > >>> wgrib2
> > > > > > > >>> command using the "-vt" argument to list the
verification
> > time.
> > > > > > > >>>
> > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that data
> > > isn't
> > > > > > > >>> valid
> > > > > > > >>> at the time you're requesting.
> > > > > > > >>>
> > > > > > > >>> John
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
> > Affiliate
> > > <
> > > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi, John,
> > > > > > > >>>>
> > > > > > > >>>> I haven't been able to work on this for awhile, but
now
> I'm
> > > > > > > >>>> working on
> > > > > > > >>>> it again but seem to be running into issues.
> > > > > > > >>>>
> > > > > > > >>>> I have a file that has both the daily max for 1-hr
ozone
> and
> > > the
> > > > > > > >>>> daily
> > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > > >>>>
> > > > > > > >>>> 9 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>> wgrib2 oz_DAY1
> > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour ave
> > > fcst:
> > > > > > > >>>> 10 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>>
> > > > > > > >>>> As you can see they have different forecast times.
But
> > when I
> > > > > > > >>>> run
> > > > > > > >>>> point_stat, I get this output:
> > > > > > > >>>>
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>>
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> matching
> > > > > > > >>>> records
> > > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > > > >>>> OZMAX8/L1
> > > > > > > >>>> found in file: oz_DAY1
> > > > > > > >>>> WARNING:
> > > > > > > >>>>
> > > > > > > >>>> Here is the relevant portion of the point_stat
config
> file:
> > > > > > > >>>>
> > > > > > > >>>> fcst = {
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> obs = {
> > > > > > > >>>> convert(x) = x * 10^9;
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> I don't remember if I had this working previously
for both
> > the
> > > > > > > >>>> 1hr
> > > > > > > >>>> and
> > > > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue
> > for
> > > > > > > >>>> the
> > > > > > > >>>> 8hr
> > > > > > > >>>> ozone. Have any thoughts?
> > > > > > > >>>>
> > > > > > > >>>> Thanks!
> > > > > > > >>>>
> > > > > > > >>>> Perry
> > > > > > > >>>>
> > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>> <
> > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> I chose option 2, and that seems to work - I have
a .stat
> > > file
> > > > > > > >>>>> with
> > > > > > > >>>>> output. Now I'll need to do some work here to see
if the
> > > > > > > >>>>> output
> > > > > > > >>>>> looks OK,
> > > > > > > >>>>> and to configure the verification so it verifies
OZMAX1
> and
> > > > > > > >>>>> OZMAX8.
> > > > > > > >>>>>
> > > > > > > >>>>> I also wonder if this would be easily modified for
PM.
> > > > > > > >>>>>
> > > > > > > >>>>> Perry
> > > > > > > >>>>>
> > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> > > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Perry,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
> summaries
> > > > > > > >>>>>> that
> > > > > > > >>>>>> are
> > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
> that
> > > are
> > > > > > > >>>>>> purportedly
> > > > > > > >>>>>> valid at 05Z. And that's the reason why you have
so
> many
> > > obs
> > > > > > > >>>>>> rejected due
> > > > > > > >>>>>> to the valid time.
> > > > > > > >>>>>>
> > > > > > > >>>>>> It's up to you guys to decide what the "correct"
way of
> > > > > > > >>>>>> handling
> > > > > > > >>>>>> this
> > > > > > > >>>>>> is... but here are a couple of options:
> > > > > > > >>>>>>
> > > > > > > >>>>>> (1) You could use the existing setup and address
this
> > logic
> > > in
> > > > > > > >>>>>> Point-Stat config file. Change the matching time
> > obs_window
> > > > > > > >>>>>> from 0 to:
> > > > > > > >>>>>>
> > > > > > > >>>>>> obs_window = {
> > > > > > > >>>>>> beg = 3600;
> > > > > > > >>>>>> end = 3600;
> > > > > > > >>>>>> }
> > > > > > > >>>>>> This tells Point-Stat to match observations which
are
> > > exactly
> > > > > > > >>>>>> 1-
> > > > > > > >>>>>> hour
> > > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > > >>>>>>
> > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary from
> > 05Z
> > > > > > > >>>>>> to
> > > > > > > >>>>>> 05Z
> > > > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting
> > at
> > > 0:
> > > > > > > >>>>>>
> > > > > > > >>>>>> time_summary = {
> > > > > > > >>>>>> flag = TRUE;
> > > > > > > >>>>>> beg = "050000";
> > > > > > > >>>>>> end = "050000";
> > > > > > > >>>>>> ...
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks,
> > > > > > > >>>>>> John
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran -
NOAA
> > > > > > > >>>>>> Affiliate
> > > > > > > >>>>>> <
> > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file,
> > but
> > > > > > > >>>>>>> it's
> > > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 4:
> > > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > observation
> > > > > > > >>>>>>> type
> > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> > > method
> > > > > > > >>>>>>> NEAREST(1),
> > > > > > > >>>>>>> using 0 pairs.
> > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thanks!
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Perry
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > > > >>>>>>> Affiliate
> > > > > > > >>>>>>> <
> > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Hi, John,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Looks like the changing of the time stamp from
_04 to
> > _05
> > > > > > > >>>>>>>> did
> > > > > > > >>>>>>>> the
> > > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> > change.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks!
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Perry
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Perry,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
> the
> > > > > > > >>>>>>>>> timestamp
> > > > > > > >>>>>>>>> of the data from that file. I think switching
the
> > > > > > > >>>>>>>>> verification time to
> > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > > initialization
> > > > > > > >>>>>>>>> times
> > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> > wgrib2
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>> MET commands
> > > > > > > >>>>>>>>> to illustrate.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
> forecast
> > > > > > > >>>>>>>>> time
> > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > > sample
> > > > > > > >>>>>>>>> data file you
> > > > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> > > wgrib2
> > > > > > > >>>>>>>>> so
> > > > > > > >>>>>>>>> I have that
> > > > > > > >>>>>>>>> funky forecast hour:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour
ave
> > > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> fcst:vt=2018070605
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
> verbosity
> > > > > > > >>>>>>>>> level
> > > > > > > >>>>>>>>> (-v
> > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data
> it
> > is
> > > > > > > >>>>>>>>> reading:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> # Read the first record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> # Read the second record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Hope that helps.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks,
> > > > > > > >>>>>>>>> John
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran
- NOAA
> > > > > > > >>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Hi, John,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> When I implemented all the changes (and also
set it
> up
> > > so
> > > > > > > >>>>>>>>>> it's
> > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > > >>>>>>>>>>
> > > > >
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > > >>>>>>>>>> -v 4
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> But then I got this error:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array()
- No
> > > > > > > >>>>>>>>>> matching
> > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > > matching
> > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> ERROR :
> > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > > > >>>>>>>>>> forecast
> > > > > > > >>>>>>>>>> data
> > > > > > > >>>>>>>>>> found! Exiting...
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output
> for
> > > the
> > > > > > > >>>>>>>>>> file
> > > > > > > >>>>>>>>>> in this case reads:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >
> > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour ave
> > > fcst:
> > > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You had
> > > > > > > >>>>>>>>>> suggested
> > > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day, but
> > > maybe
> > > > > > > >>>>>>>>>> it needs to be
> > > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Perry,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm confused. In the previous air quality
vx we
> did,
> > > we
> > > > > > > >>>>>>>>>>> used
> > > > > > > >>>>>>>>>>> observation files named:
> > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> And those files don't exist in the July data
you
> just
> > > > > > > >>>>>>>>>>> sent.
> > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> aqm.t12z.anowpm.pb.tm00
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against
> the
> > > > > > > >>>>>>>>>>> BUFR
> > > > > > > >>>>>>>>>>> or
> > > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Let me just tell you the key points for the
config
> > file
> > > > > > > >>>>>>>>>>> settings...
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass
2
> input
> > > > > > > >>>>>>>>>>> files
> > > > > > > >>>>>>>>>>> -
> > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > > >>>>>>>>>>> pb2nc \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc
\
> > > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
> time_summary
> > > > > > > >>>>>>>>>>> like
> > > > > > > >>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0;
} //
> > > > > > > >>>>>>>>>>> previous
> > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every hour
> > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> > when
> > > at
> > > > > > > >>>>>>>>>>> least
> > > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> > station
> > > > > > > >>>>>>>>>>> over the
> > > > > > > >>>>>>>>>>> defined time period. They're identified in
the
> > output
> > > by
> > > > > > > >>>>>>>>>>> the message type:
> > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst" and
> > > "obs"
> > > > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > > > >>>>>>>>>>> Instead
> > > > > > > >>>>>>>>>>> of specifying
> > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
> > > convenient
> > > > > > > >>>>>>>>>>> to specify the
> > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify the
> > > config
> > > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > > >>>>>>>>>>> using environment variables:
> > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>> obs = {
> > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway
> <
> > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
> NOAA
> > > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the
bc grids
> > and
> > > > > > > >>>>>>>>>>>>> focus on
> > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy
> files
> > > > > > > >>>>>>>>>>>>> over
> > > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > > >>>>>>>>>>>>> let him work on the files he has for now
and if
> he
> > > runs
> > > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least
the
> 24-hr
> > > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not
> > for
> > > > > > > >>>>>>>>>>>>>> bias
> > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
> > safe,
> > > > > > > >>>>>>>>>>>>>> you
> > > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia
> for
> > > John
> > > > > > > >>>>>>>>>>>>>> to do
> > > > > > > >>>>>>>>>>>>>> some testing some production files. Do
you
> think
> > I
> > > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files are
OK for
> > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that
the ops
> > > files
> > > > > > > >>>>>>>>>>>>>> should
> > > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
> > parallels.
> > > > > > > >>>>>>>>>>>>>>> We
> > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max
hour
> > labels
> > > > > > > >>>>>>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
> upgrade.
> > > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
> correct
> > > time
> > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send John
> > > over
> > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > > > >>>>>>>>>>>>>>> versions
> > > > > > > >>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have -
but
> they
> > > can
> > > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new does
> > over
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want
> to
> > > use
> > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> > Jianping
> > > > > > > >>>>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
> > management
> > > > > > > >>>>>>>>>>>>>>>> did
> > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with
> the
> > > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation.
> I
> > > > > > > >>>>>>>>>>>>>>>> think
> > > > > > > >>>>>>>>>>>>>>>> I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some
> time
> > > ago,
> > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff. I
> > > think
> > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
> entirely
> > > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and
how to
> > > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> > John
> > > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > > Gotway <
> > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away to
> > > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for
1- and
> > 8-
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> \
> > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>
> > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > > \
> > > > > > > >>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW
\
> > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1,
> > for
> > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
> region
> > > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents
> of
> > > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > > >>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> > > level:17-40
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
> > testing
> > > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> > hours
> > > 17
> > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> > better.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are
in the
> > same
> > > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell
it which
> > one
> > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > > lead_time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps
> are
> > > off
> > > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41
hour
> lead
> > > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-
hour max
> > > value
> > > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid
at 05Z
> > > while
> > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
Point-Stat
> > > config
> > > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and
was
> > fixed.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> > > but I
> > > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look
on
> dakota
> > > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute
> > 24-
> > > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped the
> > > output
> > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands to
> > count
> > > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> > that
> > > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> > 6,555
> > > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations are
> > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that
> file
> > is
> > > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
> grep
> > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated in
> > the
> > > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
> > > duplicates
> > > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason
> why
> > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> > across 2
> > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which
we
> could
> > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard Soh
> <
> > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > > (550.319
> > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check
> in
> > > the
> > > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
> > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
> content
> > > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
> inputs.
> > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files and
> > > will
> > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> > > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change
> > the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
> > currently
> > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
> Soh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to
the
> logic
> > > for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > > 24-hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > > counts:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
> currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript to
> > dump
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> > string
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is
in PB2NC
> > not
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries...
or in
> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> > > Soh <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> > and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the
obs
> > data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28
PM John
> > > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could
you
> > please
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> PB2NC.
> > > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request
> > 24-hourly
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
> end =
> > > 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
> but
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what
> > you'd
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > > "AIRNOW_MAX_240000"
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> caused
> > by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need
2 of
> > them
> > > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44
PM John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> > PrepBUFR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute a
> 24
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
> week.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40
PM
> hsoh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
> > reasonable
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
> data
> > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index
of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file was
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
> without
> > > any
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
> > headers
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> > currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value
from 0
> > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type
for
> > > summary)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable.
> > The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger than
> > > 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be
OK.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> > > Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look
on
> dakota
> > > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump
it to
> > > ascii:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > > /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> pb2nc_output.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists
> all
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But some
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced
> by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > > pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing
or
> table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Wed Jun 05 15:06:28 2019
Hi, John,
Well, it does appear that I got the same error with MET 8.1 as I did
with
the previous version:
ERROR :
ERROR : StringArray::operator[](int) const -> range check error!
ERROR :
Perry
On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry,
>
> Did you ever get this issue resolved from 1 month ago? Or should I
log on
> to WCOSS and take a look?
>
> Thanks,
> John
>
> On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> > Hi, John,
> >
> > Well, I am running the script in batch so I don't control what
node it
> > gets. I can tell you that the working directory is here:
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >
> > And the output log file is here:
> > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
> should
> > help you figure out how the thing is run.
> >
> > Thanks!
> >
> > Perry
> >
> > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Perry,
> > >
> > > We can log on to wcoss and test there instead. Yes, pb2nc does
seem to
> > be
> > > the culprit. Can you point us the wcoss node you're using and
the
> > working
> > > directory there? We basically need to recreate your call to
pb2nc to
> try
> > > to replicate the behavior.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921
>
> > > >
> > > > Hi, David, John,
> > > >
> > > > I am not running on Theia. In fact I think it would take
quite a bit
> > of
> > > > effort to transition this thing to Theia, and I'll need to be
running
> > > this
> > > > verification on prod data anyway, which is only on WCOSS.
> > > >
> > > > However, if you wish to troubleshoot on Theia, you should do
so, and
> I
> > > > think you have all you need to be able to do so, right?
> > > >
> > > > It would seem that the error on my side is in PB2NC, so I'll
probably
> > > need
> > > > to make sure I'm running what you're running.
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Perry -
> > > > > Are you running on theia?
> > > > > John suggests that we try to troubleshoot there ...
> > > > > thanks,
> > > > > David
> > > > >
> > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > > Hi Perry -
> > > > > >
> > > > > > An update on where we are on this issue:
> > > > > > Using the prepda.2019042312.nc file you provided we indeed
> > reproduce
> > > > > > the point_stat error.
> > > > > > The point_stat error is due to only a single entry the
valid time
> > > > > > table,
> > > > > > e.g. from ncdump -v hdr_vld_table
> > > > > > hdr_vld_table =
> > > > > > "20190423_050000",
> > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33, 33,
> ...
> > > > > > which are all out of range except the first.
> > > > > >
> > > > > > If I run pb2nc here with your PREPBUFR data and your
> > PB2NCConfig_aqm
> > > > > > config file, however,
> > > > > > I generate a different prepda.2019042312.nc file that runs
> through
> > > to
> > > > > > completion with point_stat.
> > > > > > Its valid time table:
> > > > > >
> > > > > > hdr_vld_table =
> > > > > > "20190423_030000",
> > > > > > "20190423_010000",
> > > > > > "20190423_020000",
> > > > > > "20190423_230000",
> > > > > > "20190423_150000",
> > > > > > "20190423_090000",
> > > > > > "20190423_070000",
> > > > > > "20190423_060000",
> > > > > > "20190423_210000",
> > > > > > "20190423_050000",
> > > > > > "20190423_170000",
> > > > > > "20190423_100000",
> > > > > > "20190423_200000",
> > > > > > "20190423_140000",
> > > > > > "20190423_130000",
> > > > > > "20190423_160000",
> > > > > > "20190423_040000",
> > > > > > "20190423_180000",
> > > > > > "20190423_080000",
> > > > > > "20190423_120000",
> > > > > > "20190423_110000",
> > > > > > "20190423_220000",
> > > > > > "20190423_190000",
> > > > > > "20190424_000000" ;
> > > > > > }
> > > > > >
> > > > > > So it seems pb2nc is running correctly here, but not on
your
> side.
> > > > > > Let me discuss with John the best way to proceed ...
> > > > > > thanks,
> > > > > > David
> > > > > >
> > > > > >
> > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov wrote:
> > > > > > > Hi, everyone,
> > > > > > >
> > > > > > > I am attaching this to this long thread regarding the
daily max
> > of
> > > > > > > ozone
> > > > > > > verification, so we remember what we were doing back
then.
> > > > > > >
> > > > > > > I am now running this again using MET 8.0, and something
seems
> to
> > > not
> > > > > > > be
> > > > > > > working when I run point_stat. I get this error when
running a
> > > daily
> > > > > > > max
> > > > > > > of ozone:
> > > > > > >
> > > > > > > ERROR :
> > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> error!
> > > > > > > ERROR :
> > > > > > >
> > > > > > > I am using my own scripts to run this using MET, so this
is not
> > > using
> > > > > > > METplus.
> > > > > > >
> > > > > > > You can find a log file of this here:
> > > > > > >
> > > >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > > It
> > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc code
is
> > working
> > > > > > > fine, I
> > > > > > > believe.
> > > > > > >
> > > > > > > The script I am using to run this is here:
> > > > > > >
> /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > > >
> > > > > > > The prepbufr files are:
> > > > > > >
> > > > > > > /com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
and
> > > > > > > /com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > > >
> > > > > > > The model file is here:
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > > >
> > > > > > >
> > > > > > > What could be causing this error?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Perry
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > > johnhg at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Great, glad that did the trick.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
> Affiliate
> > <
> > > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > > >
> > > > > > > >> Well this worked! And Jeff confirmed to me that the
8-hr
> > > average
> > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like
in the
> > 1-hr
> > > > > > > >> avg.
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> Perry
> > > > > > > >>
> > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > > >> <johnhg at ucar.edu>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Perry,
> > > > > > > >>>
> > > > > > > >>> I think the problem is in your setting of
> > > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > > >>>
> > > > > > > >>> Here's your wgrib2 output:
> > > > > > > >>> wgrib2 oz_DAY1
> > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
> ave
> > > > > > > >>> fcst:
> > > > > > > >>>
> > > > > > > >>> The "d=2018081112" tells you the model
initialization time.
> > > The
> > > > > > > >>> valid
> > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
> > valid
> > > 16
> > > > > > > >>> hours
> > > > > > > >>> later while the OZMAX8 data is valid 22 hours later.
Rerun
> > > that
> > > > > > > >>> wgrib2
> > > > > > > >>> command using the "-vt" argument to list the
verification
> > time.
> > > > > > > >>>
> > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that data
> > > isn't
> > > > > > > >>> valid
> > > > > > > >>> at the time you're requesting.
> > > > > > > >>>
> > > > > > > >>> John
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran - NOAA
> > Affiliate
> > > <
> > > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi, John,
> > > > > > > >>>>
> > > > > > > >>>> I haven't been able to work on this for awhile, but
now
> I'm
> > > > > > > >>>> working on
> > > > > > > >>>> it again but seem to be running into issues.
> > > > > > > >>>>
> > > > > > > >>>> I have a file that has both the daily max for 1-hr
ozone
> and
> > > the
> > > > > > > >>>> daily
> > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > > >>>>
> > > > > > > >>>> 9 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>> wgrib2 oz_DAY1
> > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave fcst:
> > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour ave
> > > fcst:
> > > > > > > >>>> 10 (t10a2)
> > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>> >
> > > > > > > >>>>
> > > > > > > >>>> As you can see they have different forecast times.
But
> > when I
> > > > > > > >>>> run
> > > > > > > >>>> point_stat, I get this output:
> > > > > > > >>>>
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>>
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > >>>> DEBUG 2:
> > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() - No
> matching
> > > > > > > >>>> records
> > > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING:
> > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
matching
> > > > > > > >>>> OZMAX8/L1
> > > > > > > >>>> found in file: oz_DAY1
> > > > > > > >>>> WARNING:
> > > > > > > >>>>
> > > > > > > >>>> Here is the relevant portion of the point_stat
config
> file:
> > > > > > > >>>>
> > > > > > > >>>> fcst = {
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> obs = {
> > > > > > > >>>> convert(x) = x * 10^9;
> > > > > > > >>>> field = [
> > > > > > > >>>> { name = "COPO"; level = "A1"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > > >>>> { name = "COPO"; level = "A8"; message_type =
> > > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>> ];
> > > > > > > >>>> }
> > > > > > > >>>>
> > > > > > > >>>> I don't remember if I had this working previously
for both
> > the
> > > > > > > >>>> 1hr
> > > > > > > >>>> and
> > > > > > > >>>> the 8hr ozone, but apparently there is some sort of
issue
> > for
> > > > > > > >>>> the
> > > > > > > >>>> 8hr
> > > > > > > >>>> ozone. Have any thoughts?
> > > > > > > >>>>
> > > > > > > >>>> Thanks!
> > > > > > > >>>>
> > > > > > > >>>> Perry
> > > > > > > >>>>
> > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>> <
> > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> I chose option 2, and that seems to work - I have
a .stat
> > > file
> > > > > > > >>>>> with
> > > > > > > >>>>> output. Now I'll need to do some work here to see
if the
> > > > > > > >>>>> output
> > > > > > > >>>>> looks OK,
> > > > > > > >>>>> and to configure the verification so it verifies
OZMAX1
> and
> > > > > > > >>>>> OZMAX8.
> > > > > > > >>>>>
> > > > > > > >>>>> I also wonder if this would be easily modified for
PM.
> > > > > > > >>>>>
> > > > > > > >>>>> Perry
> > > > > > > >>>>>
> > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> > > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Perry,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute time
> summaries
> > > > > > > >>>>>> that
> > > > > > > >>>>>> are
> > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
> that
> > > are
> > > > > > > >>>>>> purportedly
> > > > > > > >>>>>> valid at 05Z. And that's the reason why you have
so
> many
> > > obs
> > > > > > > >>>>>> rejected due
> > > > > > > >>>>>> to the valid time.
> > > > > > > >>>>>>
> > > > > > > >>>>>> It's up to you guys to decide what the "correct"
way of
> > > > > > > >>>>>> handling
> > > > > > > >>>>>> this
> > > > > > > >>>>>> is... but here are a couple of options:
> > > > > > > >>>>>>
> > > > > > > >>>>>> (1) You could use the existing setup and address
this
> > logic
> > > in
> > > > > > > >>>>>> Point-Stat config file. Change the matching time
> > obs_window
> > > > > > > >>>>>> from 0 to:
> > > > > > > >>>>>>
> > > > > > > >>>>>> obs_window = {
> > > > > > > >>>>>> beg = 3600;
> > > > > > > >>>>>> end = 3600;
> > > > > > > >>>>>> }
> > > > > > > >>>>>> This tells Point-Stat to match observations which
are
> > > exactly
> > > > > > > >>>>>> 1-
> > > > > > > >>>>>> hour
> > > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > > >>>>>>
> > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary from
> > 05Z
> > > > > > > >>>>>> to
> > > > > > > >>>>>> 05Z
> > > > > > > >>>>>> instead and then leave the Point-Stat obs_window
setting
> > at
> > > 0:
> > > > > > > >>>>>>
> > > > > > > >>>>>> time_summary = {
> > > > > > > >>>>>> flag = TRUE;
> > > > > > > >>>>>> beg = "050000";
> > > > > > > >>>>>> end = "050000";
> > > > > > > >>>>>> ...
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks,
> > > > > > > >>>>>> John
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran -
NOAA
> > > > > > > >>>>>> Affiliate
> > > > > > > >>>>>> <
> > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat output
file,
> > but
> > > > > > > >>>>>>> it's
> > > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 4:
> > > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > observation
> > > > > > > >>>>>>> type
> > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
interpolation
> > > method
> > > > > > > >>>>>>> NEAREST(1),
> > > > > > > >>>>>>> using 0 pairs.
> > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Thanks!
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Perry
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran -
NOAA
> > > > > > > >>>>>>> Affiliate
> > > > > > > >>>>>>> <
> > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Hi, John,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Looks like the changing of the time stamp from
_04 to
> > _05
> > > > > > > >>>>>>>> did
> > > > > > > >>>>>>>> the
> > > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> > change.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks!
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Perry
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
Gotway <
> > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Perry,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
> the
> > > > > > > >>>>>>>>> timestamp
> > > > > > > >>>>>>>>> of the data from that file. I think switching
the
> > > > > > > >>>>>>>>> verification time to
> > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix it.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > > initialization
> > > > > > > >>>>>>>>> times
> > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> > wgrib2
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>> MET commands
> > > > > > > >>>>>>>>> to illustrate.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-t),
> forecast
> > > > > > > >>>>>>>>> time
> > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > > sample
> > > > > > > >>>>>>>>> data file you
> > > > > > > >>>>>>>>> sent to me. But remember I'm still using the
"old"
> > > wgrib2
> > > > > > > >>>>>>>>> so
> > > > > > > >>>>>>>>> I have that
> > > > > > > >>>>>>>>> funky forecast hour:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624 hour
ave
> > > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> fcst:vt=2018070605
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
> verbosity
> > > > > > > >>>>>>>>> level
> > > > > > > >>>>>>>>> (-v
> > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of the
data
> it
> > is
> > > > > > > >>>>>>>>> reading:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> # Read the first record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> # Read the second record
> > > > > > > >>>>>>>>> plot_data_plane
> > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > > >>>>>>>>> ...
> > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Hope that helps.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks,
> > > > > > > >>>>>>>>> John
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry Shafran
- NOAA
> > > > > > > >>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Hi, John,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> When I implemented all the changes (and also
set it
> up
> > > so
> > > > > > > >>>>>>>>>> it's
> > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > > >>>>>>>>>>
> > > > >
>
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > > >>>>>>>>>> -v 4
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> But then I got this error:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: MetGrib2DataFile::data_plane_array()
- No
> > > > > > > >>>>>>>>>> matching
> > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > > matching
> > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > > >>>>>>>>>> WARNING:
> > > > > > > >>>>>>>>>> ERROR :
> > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
requested
> > > > > > > >>>>>>>>>> forecast
> > > > > > > >>>>>>>>>> data
> > > > > > > >>>>>>>>>> found! Exiting...
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output
> for
> > > the
> > > > > > > >>>>>>>>>> file
> > > > > > > >>>>>>>>>> in this case reads:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > >
> > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour ave
> > > fcst:
> > > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > > >>>>>>>>>>
> > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You had
> > > > > > > >>>>>>>>>> suggested
> > > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day, but
> > > maybe
> > > > > > > >>>>>>>>>> it needs to be
> > > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
elsewhere.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
Gotway <
> > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Perry,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm confused. In the previous air quality
vx we
> did,
> > > we
> > > > > > > >>>>>>>>>>> used
> > > > > > > >>>>>>>>>>> observation files named:
> > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs and
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> And those files don't exist in the July data
you
> just
> > > > > > > >>>>>>>>>>> sent.
> > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> aqm.t12z.anowpm.pb.tm00
> > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against
> the
> > > > > > > >>>>>>>>>>> BUFR
> > > > > > > >>>>>>>>>>> or
> > > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Let me just tell you the key points for the
config
> > file
> > > > > > > >>>>>>>>>>> settings...
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to pass
2
> input
> > > > > > > >>>>>>>>>>> files
> > > > > > > >>>>>>>>>>> -
> > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > > >>>>>>>>>>> pb2nc \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00 \
> > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00.nc
\
> > > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
> time_summary
> > > > > > > >>>>>>>>>>> like
> > > > > > > >>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end = 0;
} //
> > > > > > > >>>>>>>>>>> previous
> > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every hour
> > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> > when
> > > at
> > > > > > > >>>>>>>>>>> least
> > > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> > station
> > > > > > > >>>>>>>>>>> over the
> > > > > > > >>>>>>>>>>> defined time period. They're identified in
the
> > output
> > > by
> > > > > > > >>>>>>>>>>> the message type:
> > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst" and
> > > "obs"
> > > > > > > >>>>>>>>>>> sections like this for the max of the 1-hour
data.
> > > > > > > >>>>>>>>>>> Instead
> > > > > > > >>>>>>>>>>> of specifying
> > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
> > > convenient
> > > > > > > >>>>>>>>>>> to specify the
> > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify the
> > > config
> > > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > > >>>>>>>>>>> using environment variables:
> > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>> obs = {
> > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > > >>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
message_type =
> > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > > >>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John Halley
Gotway
> <
> > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
> NOAA
> > > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the
bc grids
> > and
> > > > > > > >>>>>>>>>>>>> focus on
> > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy
> files
> > > > > > > >>>>>>>>>>>>> over
> > > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > > >>>>>>>>>>>>> let him work on the files he has for now
and if
> he
> > > runs
> > > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at least
the
> 24-hr
> > > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
grids...not
> > for
> > > > > > > >>>>>>>>>>>>>> bias
> > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
> > safe,
> > > > > > > >>>>>>>>>>>>>> you
> > > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran - NOAA
> > Affiliate
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia
> for
> > > John
> > > > > > > >>>>>>>>>>>>>> to do
> > > > > > > >>>>>>>>>>>>>> some testing some production files. Do
you
> think
> > I
> > > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files are
OK for
> > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think that
the ops
> > > files
> > > > > > > >>>>>>>>>>>>>> should
> > > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about the
> > parallels.
> > > > > > > >>>>>>>>>>>>>>> We
> > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily max
hour
> > labels
> > > > > > > >>>>>>>>>>>>>>> but
> > > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19 CMAQ
> upgrade.
> > > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have the
> correct
> > > time
> > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send John
> > > over
> > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has any
other
> > > > > > > >>>>>>>>>>>>>>> versions
> > > > > > > >>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have -
but
> they
> > > can
> > > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new does
> > over
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
jeff.mcqueen <
> > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you may
want
> to
> > > use
> > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> > Jianping
> > > > > > > >>>>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it but
> > management
> > > > > > > >>>>>>>>>>>>>>>> did
> > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do so
with
> the
> > > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation.
> I
> > > > > > > >>>>>>>>>>>>>>>> think
> > > > > > > >>>>>>>>>>>>>>>> I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some
> time
> > > ago,
> > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff. I
> > > think
> > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not too
> entirely
> > > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct and
how to
> > > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> > John
> > > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > > Gotway <
> > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away to
> > > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values for
1- and
> > 8-
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>> svn-met-
dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> \
> > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>
> > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > > \
> > > > > > > >>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc PointStatConfig_AIRNOW
\
> > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1,
> > for
> > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
> region
> > > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
contents
> of
> > > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > > >>>>>>>>>>>>>>>>>
> > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour ave
fcst:
> > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1 sigma
> > > level:17-40
> > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
> > testing
> > > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> > hours
> > > 17
> > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> > better.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours are
in the
> > same
> > > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell
it which
> > one
> > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > > lead_time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps
> are
> > > off
> > > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a 41
hour
> lead
> > > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-
hour max
> > > value
> > > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is valid
at 05Z
> > > while
> > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
Point-Stat
> > > config
> > > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back 1
hour:
> > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files and
was
> > fixed.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
summaries,
> > > but I
> > > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look
on
> dakota
> > > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc to
compute
> > 24-
> > > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped the
> > > output
> > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands to
> > count
> > > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> > pb2nc_output.txt
> > > |
> > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > wc
> > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> pb2nc_output.txt |
> > > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> > that
> > > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are unique.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> > 6,555
> > > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations are
> > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that
> file
> > is
> > > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
> grep
> > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > 32.47
> > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated in
> > the
> > > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
> > > duplicates
> > > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM hsoh
> > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason
> why
> > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> > across 2
> > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from which
we
> could
> > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard Soh
> <
> > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > > (550.319
> > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30 minutes.
> > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will not
check
> in
> > > the
> > > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input files.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
> > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
> content
> > > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple BURF
> inputs.
> > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files and
> > > will
> > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John Halley
Gotway
> > > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this will
change
> > the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case not
> > currently
> > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
> Soh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
> > summary
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to SVN.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
/d3/personal/hsoh/data/pb2nc_summary/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to
the
> logic
> > > for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > > 24-hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > > counts:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
> currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript to
> > dump
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> > string
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is
in PB2NC
> > not
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time summaries...
or in
> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54 PM
John
> > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41 PM
Howard
> > > Soh <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> > and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for the
obs
> > data.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28
PM John
> > > Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today. Could
you
> > please
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> PB2NC.
> > > I
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig \
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I request
> > 24-hourly
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z, like
this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
> end =
> > > 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
> but
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is not
what
> > you'd
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > > "AIRNOW_MAX_240000"
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never referenced.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> caused
> > by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I need
2 of
> > them
> > > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to 04Z
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a look?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:44
PM John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> > PrepBUFR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute a
> 24
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
> week.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at 6:40
PM
> hsoh
> > <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script is
> > reasonable
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
> data
> > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header index
of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file was
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from pb2nc
> without
> > > any
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
> > headers
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
> NetCDF
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (7295
> > currently)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains value
from 0
> > to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message type
for
> > > summary)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
variable.
> > The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger than
> > > 3655.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to be
OK.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >
>
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM, John
Halley
> > > Gotway
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a look
on
> dakota
> > > in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to dump
it to
> > > ascii:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > > /d1/johnhg/MET/MET_development/
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> pb2nc_output.txt
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists
> all
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But some
> > of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
evidenced
> by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > > pb2nc_output.nc
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some indexing
or
> table
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Thu Jun 06 15:21:44 2019
Perry,
I was able to replicate this behavior both on WCOSS and locally. I
believe
it's a bug in PB2NC. I'm trying to narrow it down and will then hand
it
off to Howard to debug. Thanks for identifying this issue.
John
On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>
> Hi, John,
>
> Well, it does appear that I got the same error with MET 8.1 as I did
with
> the previous version:
>
> ERROR :
> ERROR : StringArray::operator[](int) const -> range check error!
> ERROR :
>
> Perry
>
> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Perry,
> >
> > Did you ever get this issue resolved from 1 month ago? Or should
I log
> on
> > to WCOSS and take a look?
> >
> > Thanks,
> > John
> >
> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Hi, John,
> > >
> > > Well, I am running the script in batch so I don't control what
node it
> > > gets. I can tell you that the working directory is here:
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > >
> > > And the output log file is here:
> > > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
> > should
> > > help you figure out how the thing is run.
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Perry,
> > > >
> > > > We can log on to wcoss and test there instead. Yes, pb2nc
does seem
> to
> > > be
> > > > the culprit. Can you point us the wcoss node you're using and
the
> > > working
> > > > directory there? We basically need to recreate your call to
pb2nc to
> > try
> > > > to replicate the behavior.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> > > > >
> > > > > Hi, David, John,
> > > > >
> > > > > I am not running on Theia. In fact I think it would take
quite a
> bit
> > > of
> > > > > effort to transition this thing to Theia, and I'll need to
be
> running
> > > > this
> > > > > verification on prod data anyway, which is only on WCOSS.
> > > > >
> > > > > However, if you wish to troubleshoot on Theia, you should do
so,
> and
> > I
> > > > > think you have all you need to be able to do so, right?
> > > > >
> > > > > It would seem that the error on my side is in PB2NC, so I'll
> probably
> > > > need
> > > > > to make sure I'm running what you're running.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Perry
> > > > >
> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Perry -
> > > > > > Are you running on theia?
> > > > > > John suggests that we try to troubleshoot there ...
> > > > > > thanks,
> > > > > > David
> > > > > >
> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> > > > > > > Hi Perry -
> > > > > > >
> > > > > > > An update on where we are on this issue:
> > > > > > > Using the prepda.2019042312.nc file you provided we
indeed
> > > reproduce
> > > > > > > the point_stat error.
> > > > > > > The point_stat error is due to only a single entry the
valid
> time
> > > > > > > table,
> > > > > > > e.g. from ncdump -v hdr_vld_table
> > > > > > > hdr_vld_table =
> > > > > > > "20190423_050000",
> > > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33, 33,
> > ...
> > > > > > > which are all out of range except the first.
> > > > > > >
> > > > > > > If I run pb2nc here with your PREPBUFR data and your
> > > PB2NCConfig_aqm
> > > > > > > config file, however,
> > > > > > > I generate a different prepda.2019042312.nc file that
runs
> > through
> > > > to
> > > > > > > completion with point_stat.
> > > > > > > Its valid time table:
> > > > > > >
> > > > > > > hdr_vld_table =
> > > > > > > "20190423_030000",
> > > > > > > "20190423_010000",
> > > > > > > "20190423_020000",
> > > > > > > "20190423_230000",
> > > > > > > "20190423_150000",
> > > > > > > "20190423_090000",
> > > > > > > "20190423_070000",
> > > > > > > "20190423_060000",
> > > > > > > "20190423_210000",
> > > > > > > "20190423_050000",
> > > > > > > "20190423_170000",
> > > > > > > "20190423_100000",
> > > > > > > "20190423_200000",
> > > > > > > "20190423_140000",
> > > > > > > "20190423_130000",
> > > > > > > "20190423_160000",
> > > > > > > "20190423_040000",
> > > > > > > "20190423_180000",
> > > > > > > "20190423_080000",
> > > > > > > "20190423_120000",
> > > > > > > "20190423_110000",
> > > > > > > "20190423_220000",
> > > > > > > "20190423_190000",
> > > > > > > "20190424_000000" ;
> > > > > > > }
> > > > > > >
> > > > > > > So it seems pb2nc is running correctly here, but not on
your
> > side.
> > > > > > > Let me discuss with John the best way to proceed ...
> > > > > > > thanks,
> > > > > > > David
> > > > > > >
> > > > > > >
> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
wrote:
> > > > > > > > Hi, everyone,
> > > > > > > >
> > > > > > > > I am attaching this to this long thread regarding the
daily
> max
> > > of
> > > > > > > > ozone
> > > > > > > > verification, so we remember what we were doing back
then.
> > > > > > > >
> > > > > > > > I am now running this again using MET 8.0, and
something
> seems
> > to
> > > > not
> > > > > > > > be
> > > > > > > > working when I run point_stat. I get this error when
> running a
> > > > daily
> > > > > > > > max
> > > > > > > > of ozone:
> > > > > > > >
> > > > > > > > ERROR :
> > > > > > > > ERROR : StringArray::operator[](int) const -> range
check
> > error!
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > I am using my own scripts to run this using MET, so
this is
> not
> > > > using
> > > > > > > > METplus.
> > > > > > > >
> > > > > > > > You can find a log file of this here:
> > > > > > > >
> > > > >
> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> > > > > > > > It
> > > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc
code is
> > > working
> > > > > > > > fine, I
> > > > > > > > believe.
> > > > > > > >
> > > > > > > > The script I am using to run this is here:
> > > > > > > >
> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> > > > > > > >
> > > > > > > > The prepbufr files are:
> > > > > > > >
> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> > > > > > > >
> > > > > > > > The model file is here:
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> > > > > > > >
> > > > > > > >
> > > > > > > > What could be causing this error?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Perry
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
> > > > johnhg at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Great, glad that did the trick.
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran - NOAA
> > Affiliate
> > > <
> > > > > > > > > perry.shafran at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > >> Well this worked! And Jeff confirmed to me that
the 8-hr
> > > > average
> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z like
in the
> > > 1-hr
> > > > > > > > >> avg.
> > > > > > > > >>
> > > > > > > > >> Thanks!
> > > > > > > > >>
> > > > > > > > >> Perry
> > > > > > > > >>
> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley Gotway
> > > > > > > > >> <johnhg at ucar.edu>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> Perry,
> > > > > > > > >>>
> > > > > > > > >>> I think the problem is in your setting of
> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
> > > > > > > > >>>
> > > > > > > > >>> Here's your wgrib2 output:
> > > > > > > > >>> wgrib2 oz_DAY1
> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour ave
> > fcst:
> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-
22 hour
> > ave
> > > > > > > > >>> fcst:
> > > > > > > > >>>
> > > > > > > > >>> The "d=2018081112" tells you the model
initialization
> time.
> > > > The
> > > > > > > > >>> valid
> > > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
> > > valid
> > > > 16
> > > > > > > > >>> hours
> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
later.
> Rerun
> > > > that
> > > > > > > > >>> wgrib2
> > > > > > > > >>> command using the "-vt" argument to list the
verification
> > > time.
> > > > > > > > >>>
> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that
> data
> > > > isn't
> > > > > > > > >>> valid
> > > > > > > > >>> at the time you're requesting.
> > > > > > > > >>>
> > > > > > > > >>> John
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran -
NOAA
> > > Affiliate
> > > > <
> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi, John,
> > > > > > > > >>>>
> > > > > > > > >>>> I haven't been able to work on this for awhile,
but now
> > I'm
> > > > > > > > >>>> working on
> > > > > > > > >>>> it again but seem to be running into issues.
> > > > > > > > >>>>
> > > > > > > > >>>> I have a file that has both the daily max for 1-
hr ozone
> > and
> > > > the
> > > > > > > > >>>> daily
> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks like
this:
> > > > > > > > >>>>
> > > > > > > > >>>> 9 (t10a2)
> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > > >>>> >
> > > > > > > > >>>> wgrib2 oz_DAY1
> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
> fcst:
> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
> ave
> > > > fcst:
> > > > > > > > >>>> 10 (t10a2)
> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > > >>>> >
> > > > > > > > >>>>
> > > > > > > > >>>> As you can see they have different forecast
times. But
> > > when I
> > > > > > > > >>>> run
> > > > > > > > >>>> point_stat, I get this output:
> > > > > > > > >>>>
> > > > > > > > >>>> DEBUG 2:
> > > > > > > > >>>> DEBUG 2:
> > > > > > > > >>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > > >>>> DEBUG 2:
> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
> > matching
> > > > > > > > >>>> records
> > > > > > > > >>>> found for 'OZMAX8/L1'
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
> matching
> > > > > > > > >>>> OZMAX8/L1
> > > > > > > > >>>> found in file: oz_DAY1
> > > > > > > > >>>> WARNING:
> > > > > > > > >>>>
> > > > > > > > >>>> Here is the relevant portion of the point_stat
config
> > file:
> > > > > > > > >>>>
> > > > > > > > >>>> fcst = {
> > > > > > > > >>>> field = [
> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > > >>>> ];
> > > > > > > > >>>> }
> > > > > > > > >>>>
> > > > > > > > >>>> obs = {
> > > > > > > > >>>> convert(x) = x * 10^9;
> > > > > > > > >>>> field = [
> > > > > > > > >>>> { name = "COPO"; level = "A1"; message_type
=
> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
> > > > > > > > >>>> { name = "COPO"; level = "A8"; message_type
=
> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
> > > > > > > > >>>> ];
> > > > > > > > >>>> }
> > > > > > > > >>>>
> > > > > > > > >>>> I don't remember if I had this working previously
for
> both
> > > the
> > > > > > > > >>>> 1hr
> > > > > > > > >>>> and
> > > > > > > > >>>> the 8hr ozone, but apparently there is some sort
of
> issue
> > > for
> > > > > > > > >>>> the
> > > > > > > > >>>> 8hr
> > > > > > > > >>>> ozone. Have any thoughts?
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks!
> > > > > > > > >>>>
> > > > > > > > >>>> Perry
> > > > > > > > >>>>
> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
> > > > Affiliate
> > > > > > > > >>>> <
> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> I chose option 2, and that seems to work - I
have a
> .stat
> > > > file
> > > > > > > > >>>>> with
> > > > > > > > >>>>> output. Now I'll need to do some work here to
see if
> the
> > > > > > > > >>>>> output
> > > > > > > > >>>>> looks OK,
> > > > > > > > >>>>> and to configure the verification so it verifies
OZMAX1
> > and
> > > > > > > > >>>>> OZMAX8.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I also wonder if this would be easily modified
for PM.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Perry
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> > > > > > > > >>>>> <johnhg at ucar.edu>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Perry,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute
time
> > summaries
> > > > > > > > >>>>>> that
> > > > > > > > >>>>>> are
> > > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
> > that
> > > > are
> > > > > > > > >>>>>> purportedly
> > > > > > > > >>>>>> valid at 05Z. And that's the reason why you
have so
> > many
> > > > obs
> > > > > > > > >>>>>> rejected due
> > > > > > > > >>>>>> to the valid time.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> It's up to you guys to decide what the
"correct" way
> of
> > > > > > > > >>>>>> handling
> > > > > > > > >>>>>> this
> > > > > > > > >>>>>> is... but here are a couple of options:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> (1) You could use the existing setup and
address this
> > > logic
> > > > in
> > > > > > > > >>>>>> Point-Stat config file. Change the matching
time
> > > obs_window
> > > > > > > > >>>>>> from 0 to:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> obs_window = {
> > > > > > > > >>>>>> beg = 3600;
> > > > > > > > >>>>>> end = 3600;
> > > > > > > > >>>>>> }
> > > > > > > > >>>>>> This tells Point-Stat to match observations
which are
> > > > exactly
> > > > > > > > >>>>>> 1-
> > > > > > > > >>>>>> hour
> > > > > > > > >>>>>> prior to the valid time of the forecast.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary
> from
> > > 05Z
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>> 05Z
> > > > > > > > >>>>>> instead and then leave the Point-Stat
obs_window
> setting
> > > at
> > > > 0:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> time_summary = {
> > > > > > > > >>>>>> flag = TRUE;
> > > > > > > > >>>>>> beg = "050000";
> > > > > > > > >>>>>> end = "050000";
> > > > > > > > >>>>>> ...
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks,
> > > > > > > > >>>>>> John
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran
- NOAA
> > > > > > > > >>>>>> Affiliate
> > > > > > > > >>>>>> <
> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
output
> file,
> > > but
> > > > > > > > >>>>>>> it's
> > > > > > > > >>>>>>> empty. Here's the output from point_stat:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> DEBUG 4:
> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
> > > > observation
> > > > > > > > >>>>>>> type
> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
> interpolation
> > > > method
> > > > > > > > >>>>>>> NEAREST(1),
> > > > > > > > >>>>>>> using 0 pairs.
> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> > > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Have any thoughts about this one?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thanks!
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Perry
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry Shafran
- NOAA
> > > > > > > > >>>>>>> Affiliate
> > > > > > > > >>>>>>> <
> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Hi, John,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Looks like the changing of the time stamp
from _04
> to
> > > _05
> > > > > > > > >>>>>>>> did
> > > > > > > > >>>>>>>> the
> > > > > > > > >>>>>>>> trick as point_stat ran without an issue with
that
> > > change.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thanks!
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Perry
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John Halley
> Gotway <
> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Perry,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
> > the
> > > > > > > > >>>>>>>>> timestamp
> > > > > > > > >>>>>>>>> of the data from that file. I think
switching the
> > > > > > > > >>>>>>>>> verification time to
> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix
it.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
> > > > initialization
> > > > > > > > >>>>>>>>> times
> > > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
> > > wgrib2
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>> MET commands
> > > > > > > > >>>>>>>>> to illustrate.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-
t),
> > forecast
> > > > > > > > >>>>>>>>> time
> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using a
> > > > sample
> > > > > > > > >>>>>>>>> data file you
> > > > > > > > >>>>>>>>> sent to me. But remember I'm still using
the "old"
> > > > wgrib2
> > > > > > > > >>>>>>>>> so
> > > > > > > > >>>>>>>>> I have that
> > > > > > > > >>>>>>>>> funky forecast hour:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> > > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624
hour ave
> > > > > > > > >>>>>>>>> fcst:vt=2018070505
> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> > fcst:vt=2018070605
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a higher
> > verbosity
> > > > > > > > >>>>>>>>> level
> > > > > > > > >>>>>>>>> (-v
> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of
the data
> > it
> > > is
> > > > > > > > >>>>>>>>> reading:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> # Read the first record
> > > > > > > > >>>>>>>>> plot_data_plane
> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> > > > > > > > >>>>>>>>> ...
> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > > >>>>>>>>> ...
> > > > > > > > >>>>>>>>> # Read the second record
> > > > > > > > >>>>>>>>> plot_data_plane
> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> > > > > > > > >>>>>>>>> ...
> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Hope that helps.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>> John
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
Shafran -
> NOAA
> > > > > > > > >>>>>>>>> Affiliate <
> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Hi, John,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> When I implemented all the changes (and
also set
> it
> > up
> > > > so
> > > > > > > > >>>>>>>>>> it's
> > > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
> > > > > > > > >>>>>>>>>>
> > > > > >
> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> > > > > > > > >>>>>>>>>> -v 4
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> But then I got this error:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> DEBUG 2:
> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> WARNING:
MetGrib2DataFile::data_plane_array() - No
> > > > > > > > >>>>>>>>>> matching
> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
> > > > matching
> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> > > > > > > > >>>>>>>>>> WARNING:
> > > > > > > > >>>>>>>>>> ERROR :
> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
> requested
> > > > > > > > >>>>>>>>>> forecast
> > > > > > > > >>>>>>>>>> data
> > > > > > > > >>>>>>>>>> found! Exiting...
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the wgrib2
output
> > for
> > > > the
> > > > > > > > >>>>>>>>>> file
> > > > > > > > >>>>>>>>>> in this case reads:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> 14 (t14a2)
> > > > > > > > >>>>>>>>>>
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > >
> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-16
hour
> ave
> > > > fcst:
> > > > > > > > >>>>>>>>>> 15 (t14a2)
> > > > > > > > >>>>>>>>>>
> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You
> had
> > > > > > > > >>>>>>>>>> suggested
> > > > > > > > >>>>>>>>>> that it should be the time stamp of the 2nd
day,
> but
> > > > maybe
> > > > > > > > >>>>>>>>>> it needs to be
> > > > > > > > >>>>>>>>>> the time stamp of the first day?
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
> elsewhere.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John Halley
> Gotway <
> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Perry,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air quality
vx we
> > did,
> > > > we
> > > > > > > > >>>>>>>>>>> used
> > > > > > > > >>>>>>>>>>> observation files named:
> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs
and
> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> And those files don't exist in the July
data you
> > just
> > > > > > > > >>>>>>>>>>> sent.
> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> > aqm.t12z.anowpm.pb.tm00
> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
aqm.t12z.anowpm.pb.tm120
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
against
> > the
> > > > > > > > >>>>>>>>>>> BUFR
> > > > > > > > >>>>>>>>>>> or
> > > > > > > > >>>>>>>>>>> PREPBUFR.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Let me just tell you the key points for
the
> config
> > > file
> > > > > > > > >>>>>>>>>>> settings...
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to
pass 2
> > input
> > > > > > > > >>>>>>>>>>> files
> > > > > > > > >>>>>>>>>>> -
> > > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> > > > > > > > >>>>>>>>>>> pb2nc \
> > > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00
\
> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
> > > > > > > > >>>>>>>>>>> PB2NCConfig \
> > > > > > > > >>>>>>>>>>> -pbfile
hourly.20180704/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
> > time_summary
> > > > > > > > >>>>>>>>>>> like
> > > > > > > > >>>>>>>>>>> this:
> > > > > > > > >>>>>>>>>>> time_summary = {
> > > > > > > > >>>>>>>>>>> flag = TRUE;
> > > > > > > > >>>>>>>>>>> beg = "040000";
> > > > > > > > >>>>>>>>>>> end = "040000";
> > > > > > > > >>>>>>>>>>> step = 24*3600;
> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end =
0; } //
> > > > > > > > >>>>>>>>>>> previous
> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> > > > > > > > >>>>>>>>>>> grib_code = [];
> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every
> hour
> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
summaries
> > > when
> > > > at
> > > > > > > > >>>>>>>>>>> least
> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
> > > > > > > > >>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values at
each
> > > station
> > > > > > > > >>>>>>>>>>> over the
> > > > > > > > >>>>>>>>>>> defined time period. They're identified
in the
> > > output
> > > > by
> > > > > > > > >>>>>>>>>>> the message type:
> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst"
> and
> > > > "obs"
> > > > > > > > >>>>>>>>>>> sections like this for the max of the 1-
hour
> data.
> > > > > > > > >>>>>>>>>>> Instead
> > > > > > > > >>>>>>>>>>> of specifying
> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
> > > > convenient
> > > > > > > > >>>>>>>>>>> to specify the
> > > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify the
> > > > config
> > > > > > > > >>>>>>>>>>> file, I'd suggest
> > > > > > > > >>>>>>>>>>> using environment variables:
> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> fcst = {
> > > > > > > > >>>>>>>>>>> field = [
> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> > > > > > > > >>>>>>>>>>> ];
> > > > > > > > >>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>> obs = {
> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> > > > > > > > >>>>>>>>>>> field = [
> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
> message_type =
> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> > > > > > > > >>>>>>>>>>> ];
> > > > > > > > >>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
Halley
> Gotway
> > <
> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
> > NOAA
> > > > > > > > >>>>>>>>>>>> Affiliate <
> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all the
bc
> grids
> > > and
> > > > > > > > >>>>>>>>>>>>> focus on
> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time to
copy
> > files
> > > > > > > > >>>>>>>>>>>>> over
> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
> > > > > > > > >>>>>>>>>>>>> let him work on the files he has for now
and if
> > he
> > > > runs
> > > > > > > > >>>>>>>>>>>>> into problems, I'll
> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
least the
> > 24-hr
> > > > > > > > >>>>>>>>>>>>> max/avg files).
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Thanks!
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
> grids...not
> > > for
> > > > > > > > >>>>>>>>>>>>>> bias
> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
> > > safe,
> > > > > > > > >>>>>>>>>>>>>> you
> > > > > > > > >>>>>>>>>>>>>> may want to provide
> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran -
NOAA
> > > Affiliate
> > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over to
Theia
> > for
> > > > John
> > > > > > > > >>>>>>>>>>>>>> to do
> > > > > > > > >>>>>>>>>>>>>> some testing some production files. Do
you
> > think
> > > I
> > > > > > > > >>>>>>>>>>>>>> should use parallels
> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files
are OK
> for
> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> > > > > > > > >>>>>>>>>>>>>> for AQ?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think
that the
> ops
> > > > files
> > > > > > > > >>>>>>>>>>>>>> should
> > > > > > > > >>>>>>>>>>>>>> be OK for now.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen <
> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about
the
> > > parallels.
> > > > > > > > >>>>>>>>>>>>>>> We
> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily
max hour
> > > labels
> > > > > > > > >>>>>>>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>>>>> management did not
> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19
CMAQ
> > upgrade.
> > > > > > > > >>>>>>>>>>>>>>> Jeff
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
> > > > Affiliate
> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have
the
> > correct
> > > > time
> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send
> John
> > > > over
> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has
any other
> > > > > > > > >>>>>>>>>>>>>>> versions
> > > > > > > > >>>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have
- but
> > they
> > > > can
> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new
> does
> > > over
> > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Thanks!
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
> jeff.mcqueen <
> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you
may want
> > to
> > > > use
> > > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> > > Jianping
> > > > > > > > >>>>>>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it
but
> > > management
> > > > > > > > >>>>>>>>>>>>>>>> did
> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do
so with
> > the
> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
> > > > > > > > >>>>>>>>>>>>>>>> implementation.
> > > > > > > > >>>>>>>>>>>>>>>> Jeff
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
> > > > Affiliate
> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
conversation.
> > I
> > > > > > > > >>>>>>>>>>>>>>>> think
> > > > > > > > >>>>>>>>>>>>>>>> I
> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the timestamps
some
> > time
> > > > ago,
> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
Jeff.
> I
> > > > think
> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not
too
> > entirely
> > > > > > > > >>>>>>>>>>>>>>>> sure.
> > > > > > > > >>>>>>>>>>>>>>>> But he would have
> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct
and how
> to
> > > > > > > > >>>>>>>>>>>>>>>> approach
> > > > > > > > >>>>>>>>>>>>>>>> this.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time stamp
issues
> > > John
> > > > > > > > >>>>>>>>>>>>>>>> talks
> > > > > > > > >>>>>>>>>>>>>>>> about?
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Perry
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
> > > > Gotway <
> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all squared
away
> to
> > > > > > > > >>>>>>>>>>>>>>>>> compute
> > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values
for 1-
> and
> > > 8-
> > > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>
> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > \
> > > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like this:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> > > > > \
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
PointStatConfig_AIRNOW \
> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
> COPO/A1,
> > > for
> > > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
> > region
> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147 pairs.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
> contents
> > of
> > > > > > > > >>>>>>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>>> "max"
> > > > > > > > >>>>>>>>>>>>>>>>> file:
> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour
ave
> fcst:
> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
sigma
> > > > level:17-40
> > > > > > > > >>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
> > > testing
> > > > > > > > >>>>>>>>>>>>>>>>> revealed
> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
> > > hours
> > > > 17
> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> > > better.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours
are in
> the
> > > same
> > > > > > > > >>>>>>>>>>>>>>>>> file, we
> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to tell
it
> which
> > > one
> > > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> > > > > > > > >>>>>>>>>>>>>>>>> field = [
> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
> > > > lead_time
> > > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> > > > > > > > >>>>>>>>>>>>>>>>> ];
> > > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
timestamps
> > are
> > > > off
> > > > > > > > >>>>>>>>>>>>>>>>> by 1
> > > > > > > > >>>>>>>>>>>>>>>>> hour.
> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a
41 hour
> > lead
> > > > > > > > >>>>>>>>>>>>>>>>> time
> > > > > > > > >>>>>>>>>>>>>>>>> =
> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a 24-
hour
> max
> > > > value
> > > > > > > > >>>>>>>>>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
valid at
> 05Z
> > > > while
> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
Point-Stat
> > > > config
> > > > > > > > >>>>>>>>>>>>>>>>> file by
> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look back
1
> hour:
> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> > > > > > > > >>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM hsoh
> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files
and was
> > > fixed.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
> > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
> summaries,
> > > > but I
> > > > > > > > >>>>>>>>>>>>>>>>>> found
> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a look
on
> > dakota
> > > > > > > > >>>>>>>>>>>>>>>>>> here:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc
to
> compute
> > > 24-
> > > > > > > > >>>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped
> the
> > > > output
> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands to
> > > count
> > > > > > > > >>>>>>>>>>>>>>>>>> up
> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> > > pb2nc_output.txt
> > > > |
> > > > > > > > >>>>>>>>>>>>>>>>>> wc
> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> > > pb2nc_output.txt
> > > > |
> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> > pb2nc_output.txt |
> > > > wc
> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> > pb2nc_output.txt |
> > > > > > > > >>>>>>>>>>>>>>>>>> sort
> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
meaning
> > > that
> > > > > > > > >>>>>>>>>>>>>>>>>> all
> > > > > > > > >>>>>>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
unique.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
> > > 6,555
> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations
> are
> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID in
that
> > file
> > > is
> > > > > > > > >>>>>>>>>>>>>>>>>> named
> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
> > grep
> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180527_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
20180528_040000
> > > 32.47
> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
repeated
> in
> > > the
> > > > > > > > >>>>>>>>>>>>>>>>>> last two
> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
> > > > duplicates
> > > > > > > > >>>>>>>>>>>>>>>>>> are
> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM
hsoh
> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
> > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's no
reason
> > why
> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> > > > > > > > >>>>>>>>>>>>>>>>>>> have
> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> > > across 2
> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from
which we
> > could
> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard
> Soh
> > <
> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
> > > > (550.319
> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
minutes.
> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will
not check
> > in
> > > > the
> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
files.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
> > content
> > > > of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
> > > summary
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple
BURF
> > inputs.
> > > I
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files
> and
> > > > will
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
Halley
> Gotway
> > > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this
will
> change
> > > the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case
not
> > > currently
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
> > Soh
> > > <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
> > > summary
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to
SVN.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
> /d3/personal/hsoh/data/pb2nc_summary/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52 PM
John
> > > Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates to
the
> > logic
> > > > for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
compute
> > > > 24-hour
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
> > > > counts:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
> > currently)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript
> to
> > > dump
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
> > > string
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem is
in
> PB2NC
> > > not
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
summaries... or in
> > the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54
PM John
> > > Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41
PM
> Howard
> > > > Soh <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291 observations
from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
aqm.20180527/aqm.t12z.prepbufr.tm00
> > > and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for
the obs
> > > data.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28
PM
> John
> > > > Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
Could you
> > > please
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> > PB2NC.
> > > > I
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc PB2NCConfig
\
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
request
> > > 24-hourly
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z,
like
> this:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
> > end =
> > > > 0;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
> > but
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is
not what
> > > you'd
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> > > > "AIRNOW_MAX_240000"
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
referenced.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> > caused
> > > by
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I
need 2 of
> > > them
> > > > to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to
04Z
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a
look?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:44 PM
> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> > > PrepBUFR
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
compute
> a
> > 24
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
> > week.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:40 PM
> > hsoh
> > > <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script
is
> > > reasonable
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
> > data
> > > in
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
index of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file was
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from
pb2nc
> > without
> > > > any
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
> > > headers
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
> > NetCDF
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(7295
> > > currently)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
value
> from 0
> > > to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message
type for
> > > > summary)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
> variable.
> > > The
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger
> than
> > > > 3655.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to
be OK.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >
> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEDIAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEDIAN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM,
John
> Halley
> > > > Gotway
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a
look on
> > dakota
> > > > in
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to
dump it
> to
> > > > ascii:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> > > > /d1/johnhg/MET/MET_development/
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> > pb2nc_output.txt
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error but
lists
> > all
> > > of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But
> some
> > > of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
> evidenced
> > by
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
> > > > pb2nc_output.nc
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
indexing or
> > table
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: John Halley Gotway
Time: Thu Jun 06 16:21:19 2019
Perry (and Howard),
I isolated this issue down and wrote up a GitHub issue describing the
problem:
https://github.com/NCAR/MET/issues/1137
Howard, please take a look and let me know if you need any help
further
debugging this.
Thanks,
John
On Thu, Jun 6, 2019 at 3:21 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Perry,
>
> I was able to replicate this behavior both on WCOSS and locally. I
> believe it's a bug in PB2NC. I'm trying to narrow it down and will
then
> hand it off to Howard to debug. Thanks for identifying this issue.
>
> John
>
> On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>>
>> Hi, John,
>>
>> Well, it does appear that I got the same error with MET 8.1 as I
did with
>> the previous version:
>>
>> ERROR :
>> ERROR : StringArray::operator[](int) const -> range check error!
>> ERROR :
>>
>> Perry
>>
>> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Perry,
>> >
>> > Did you ever get this issue resolved from 1 month ago? Or should
I log
>> on
>> > to WCOSS and take a look?
>> >
>> > Thanks,
>> > John
>> >
>> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Hi, John,
>> > >
>> > > Well, I am running the script in batch so I don't control what
node it
>> > > gets. I can tell you that the working directory is here:
>> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > >
>> > > And the output log file is here:
>> > > /ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688,
which
>> > should
>> > > help you figure out how the thing is run.
>> > >
>> > > Thanks!
>> > >
>> > > Perry
>> > >
>> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
>> > > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Perry,
>> > > >
>> > > > We can log on to wcoss and test there instead. Yes, pb2nc
does
>> seem to
>> > > be
>> > > > the culprit. Can you point us the wcoss node you're using
and the
>> > > working
>> > > > directory there? We basically need to recreate your call to
pb2nc
>> to
>> > try
>> > > > to replicate the behavior.
>> > > >
>> > > > Thanks,
>> > > > John
>> > > >
>> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via RT
<
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
>> > > > >
>> > > > > Hi, David, John,
>> > > > >
>> > > > > I am not running on Theia. In fact I think it would take
quite a
>> bit
>> > > of
>> > > > > effort to transition this thing to Theia, and I'll need to
be
>> running
>> > > > this
>> > > > > verification on prod data anyway, which is only on WCOSS.
>> > > > >
>> > > > > However, if you wish to troubleshoot on Theia, you should
do so,
>> and
>> > I
>> > > > > think you have all you need to be able to do so, right?
>> > > > >
>> > > > > It would seem that the error on my side is in PB2NC, so
I'll
>> probably
>> > > > need
>> > > > > to make sure I'm running what you're running.
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > Perry
>> > > > >
>> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
>> > > met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Perry -
>> > > > > > Are you running on theia?
>> > > > > > John suggests that we try to troubleshoot there ...
>> > > > > > thanks,
>> > > > > > David
>> > > > > >
>> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
>> > > > > > > Hi Perry -
>> > > > > > >
>> > > > > > > An update on where we are on this issue:
>> > > > > > > Using the prepda.2019042312.nc file you provided we
indeed
>> > > reproduce
>> > > > > > > the point_stat error.
>> > > > > > > The point_stat error is due to only a single entry the
valid
>> time
>> > > > > > > table,
>> > > > > > > e.g. from ncdump -v hdr_vld_table
>> > > > > > > hdr_vld_table =
>> > > > > > > "20190423_050000",
>> > > > > > > while the hdr_vld indices in that file are hdr_vld = 0,
33,
>> 33,
>> > ...
>> > > > > > > which are all out of range except the first.
>> > > > > > >
>> > > > > > > If I run pb2nc here with your PREPBUFR data and your
>> > > PB2NCConfig_aqm
>> > > > > > > config file, however,
>> > > > > > > I generate a different prepda.2019042312.nc file that
runs
>> > through
>> > > > to
>> > > > > > > completion with point_stat.
>> > > > > > > Its valid time table:
>> > > > > > >
>> > > > > > > hdr_vld_table =
>> > > > > > > "20190423_030000",
>> > > > > > > "20190423_010000",
>> > > > > > > "20190423_020000",
>> > > > > > > "20190423_230000",
>> > > > > > > "20190423_150000",
>> > > > > > > "20190423_090000",
>> > > > > > > "20190423_070000",
>> > > > > > > "20190423_060000",
>> > > > > > > "20190423_210000",
>> > > > > > > "20190423_050000",
>> > > > > > > "20190423_170000",
>> > > > > > > "20190423_100000",
>> > > > > > > "20190423_200000",
>> > > > > > > "20190423_140000",
>> > > > > > > "20190423_130000",
>> > > > > > > "20190423_160000",
>> > > > > > > "20190423_040000",
>> > > > > > > "20190423_180000",
>> > > > > > > "20190423_080000",
>> > > > > > > "20190423_120000",
>> > > > > > > "20190423_110000",
>> > > > > > > "20190423_220000",
>> > > > > > > "20190423_190000",
>> > > > > > > "20190424_000000" ;
>> > > > > > > }
>> > > > > > >
>> > > > > > > So it seems pb2nc is running correctly here, but not on
your
>> > side.
>> > > > > > > Let me discuss with John the best way to proceed ...
>> > > > > > > thanks,
>> > > > > > > David
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
wrote:
>> > > > > > > > Hi, everyone,
>> > > > > > > >
>> > > > > > > > I am attaching this to this long thread regarding the
daily
>> max
>> > > of
>> > > > > > > > ozone
>> > > > > > > > verification, so we remember what we were doing back
then.
>> > > > > > > >
>> > > > > > > > I am now running this again using MET 8.0, and
something
>> seems
>> > to
>> > > > not
>> > > > > > > > be
>> > > > > > > > working when I run point_stat. I get this error when
>> running a
>> > > > daily
>> > > > > > > > max
>> > > > > > > > of ozone:
>> > > > > > > >
>> > > > > > > > ERROR :
>> > > > > > > > ERROR : StringArray::operator[](int) const -> range
check
>> > error!
>> > > > > > > > ERROR :
>> > > > > > > >
>> > > > > > > > I am using my own scripts to run this using MET, so
this is
>> not
>> > > > using
>> > > > > > > > METplus.
>> > > > > > > >
>> > > > > > > > You can find a log file of this here:
>> > > > > > > >
>> > > > >
>> >
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
>> > > > > > > > It
>> > > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc
code is
>> > > working
>> > > > > > > > fine, I
>> > > > > > > > believe.
>> > > > > > > >
>> > > > > > > > The script I am using to run this is here:
>> > > > > > > >
>> > /meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
>> > > > > > > >
>> > > > > > > > The prepbufr files are:
>> > > > > > > >
>> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00 and
>> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
>> > > > > > > >
>> > > > > > > > The model file is here:
>> > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > What could be causing this error?
>> > > > > > > >
>> > > > > > > > Thanks!
>> > > > > > > >
>> > > > > > > > Perry
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway <
>> > > > johnhg at ucar.edu>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Great, glad that did the trick.
>> > > > > > > > >
>> > > > > > > > > John
>> > > > > > > > >
>> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran -
NOAA
>> > Affiliate
>> > > <
>> > > > > > > > > perry.shafran at noaa.gov> wrote:
>> > > > > > > > >
>> > > > > > > > >> Well this worked! And Jeff confirmed to me that
the 8-hr
>> > > > average
>> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z
like in
>> the
>> > > 1-hr
>> > > > > > > > >> avg.
>> > > > > > > > >>
>> > > > > > > > >> Thanks!
>> > > > > > > > >>
>> > > > > > > > >> Perry
>> > > > > > > > >>
>> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley
Gotway
>> > > > > > > > >> <johnhg at ucar.edu>
>> > > > > > > > >> wrote:
>> > > > > > > > >>
>> > > > > > > > >>> Perry,
>> > > > > > > > >>>
>> > > > > > > > >>> I think the problem is in your setting of
>> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
>> > > > > > > > >>>
>> > > > > > > > >>> Here's your wgrib2 output:
>> > > > > > > > >>> wgrib2 oz_DAY1
>> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour ave
>> > fcst:
>> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-
22 hour
>> > ave
>> > > > > > > > >>> fcst:
>> > > > > > > > >>>
>> > > > > > > > >>> The "d=2018081112" tells you the model
initialization
>> time.
>> > > > The
>> > > > > > > > >>> valid
>> > > > > > > > >>> time = init time + forecast time. So the OZMAX1
data is
>> > > valid
>> > > > 16
>> > > > > > > > >>> hours
>> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
later.
>> Rerun
>> > > > that
>> > > > > > > > >>> wgrib2
>> > > > > > > > >>> command using the "-vt" argument to list the
>> verification
>> > > time.
>> > > > > > > > >>>
>> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data because
that
>> data
>> > > > isn't
>> > > > > > > > >>> valid
>> > > > > > > > >>> at the time you're requesting.
>> > > > > > > > >>>
>> > > > > > > > >>> John
>> > > > > > > > >>>
>> > > > > > > > >>>
>> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran -
NOAA
>> > > Affiliate
>> > > > <
>> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>
>> > > > > > > > >>>> Hi, John,
>> > > > > > > > >>>>
>> > > > > > > > >>>> I haven't been able to work on this for awhile,
but now
>> > I'm
>> > > > > > > > >>>> working on
>> > > > > > > > >>>> it again but seem to be running into issues.
>> > > > > > > > >>>>
>> > > > > > > > >>>> I have a file that has both the daily max for 1-
hr
>> ozone
>> > and
>> > > > the
>> > > > > > > > >>>> daily
>> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks
like this:
>> > > > > > > > >>>>
>> > > > > > > > >>>> 9 (t10a2)
>> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > > > > > >>>> >
>> > > > > > > > >>>> wgrib2 oz_DAY1
>> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16 hour
ave
>> fcst:
>> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-22
hour
>> ave
>> > > > fcst:
>> > > > > > > > >>>> 10 (t10a2)
>> > > > > > > > >>>>
/stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > > > > > >>>> >
>> > > > > > > > >>>>
>> > > > > > > > >>>> As you can see they have different forecast
times. But
>> > > when I
>> > > > > > > > >>>> run
>> > > > > > > > >>>> point_stat, I get this output:
>> > > > > > > > >>>>
>> > > > > > > > >>>> DEBUG 2:
>> > > > > > > > >>>> DEBUG 2:
>> > > > > > > > >>>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------
>> > > > > > > > >>>> DEBUG 2:
>> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array() -
No
>> > matching
>> > > > > > > > >>>> records
>> > > > > > > > >>>> found for 'OZMAX8/L1'
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no fields
>> matching
>> > > > > > > > >>>> OZMAX8/L1
>> > > > > > > > >>>> found in file: oz_DAY1
>> > > > > > > > >>>> WARNING:
>> > > > > > > > >>>>
>> > > > > > > > >>>> Here is the relevant portion of the point_stat
config
>> > file:
>> > > > > > > > >>>>
>> > > > > > > > >>>> fcst = {
>> > > > > > > > >>>> field = [
>> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
>> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
>> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
>> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
>> > > > > > > > >>>> ];
>> > > > > > > > >>>> }
>> > > > > > > > >>>>
>> > > > > > > > >>>> obs = {
>> > > > > > > > >>>> convert(x) = x * 10^9;
>> > > > > > > > >>>> field = [
>> > > > > > > > >>>> { name = "COPO"; level = "A1";
message_type =
>> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
>> > > > > > > > >>>> { name = "COPO"; level = "A8";
message_type =
>> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
>> > > > > > > > >>>> ];
>> > > > > > > > >>>> }
>> > > > > > > > >>>>
>> > > > > > > > >>>> I don't remember if I had this working
previously for
>> both
>> > > the
>> > > > > > > > >>>> 1hr
>> > > > > > > > >>>> and
>> > > > > > > > >>>> the 8hr ozone, but apparently there is some sort
of
>> issue
>> > > for
>> > > > > > > > >>>> the
>> > > > > > > > >>>> 8hr
>> > > > > > > > >>>> ozone. Have any thoughts?
>> > > > > > > > >>>>
>> > > > > > > > >>>> Thanks!
>> > > > > > > > >>>>
>> > > > > > > > >>>> Perry
>> > > > > > > > >>>>
>> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran -
NOAA
>> > > > Affiliate
>> > > > > > > > >>>> <
>> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>
>> > > > > > > > >>>>> I chose option 2, and that seems to work - I
have a
>> .stat
>> > > > file
>> > > > > > > > >>>>> with
>> > > > > > > > >>>>> output. Now I'll need to do some work here to
see if
>> the
>> > > > > > > > >>>>> output
>> > > > > > > > >>>>> looks OK,
>> > > > > > > > >>>>> and to configure the verification so it
verifies
>> OZMAX1
>> > and
>> > > > > > > > >>>>> OZMAX8.
>> > > > > > > > >>>>>
>> > > > > > > > >>>>> I also wonder if this would be easily modified
for PM.
>> > > > > > > > >>>>>
>> > > > > > > > >>>>> Perry
>> > > > > > > > >>>>>
>> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
>> > > > > > > > >>>>> <johnhg at ucar.edu>
>> > > > > > > > >>>>> wrote:
>> > > > > > > > >>>>>
>> > > > > > > > >>>>>> Perry,
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute
time
>> > summaries
>> > > > > > > > >>>>>> that
>> > > > > > > > >>>>>> are
>> > > > > > > > >>>>>> valid at 04Z. And you're comparing them to
forecasts
>> > that
>> > > > are
>> > > > > > > > >>>>>> purportedly
>> > > > > > > > >>>>>> valid at 05Z. And that's the reason why you
have so
>> > many
>> > > > obs
>> > > > > > > > >>>>>> rejected due
>> > > > > > > > >>>>>> to the valid time.
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> It's up to you guys to decide what the
"correct" way
>> of
>> > > > > > > > >>>>>> handling
>> > > > > > > > >>>>>> this
>> > > > > > > > >>>>>> is... but here are a couple of options:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> (1) You could use the existing setup and
address this
>> > > logic
>> > > > in
>> > > > > > > > >>>>>> Point-Stat config file. Change the matching
time
>> > > obs_window
>> > > > > > > > >>>>>> from 0 to:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> obs_window = {
>> > > > > > > > >>>>>> beg = 3600;
>> > > > > > > > >>>>>> end = 3600;
>> > > > > > > > >>>>>> }
>> > > > > > > > >>>>>> This tells Point-Stat to match observations
which are
>> > > > exactly
>> > > > > > > > >>>>>> 1-
>> > > > > > > > >>>>>> hour
>> > > > > > > > >>>>>> prior to the valid time of the forecast.
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary
>> from
>> > > 05Z
>> > > > > > > > >>>>>> to
>> > > > > > > > >>>>>> 05Z
>> > > > > > > > >>>>>> instead and then leave the Point-Stat
obs_window
>> setting
>> > > at
>> > > > 0:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> time_summary = {
>> > > > > > > > >>>>>> flag = TRUE;
>> > > > > > > > >>>>>> beg = "050000";
>> > > > > > > > >>>>>> end = "050000";
>> > > > > > > > >>>>>> ...
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> Thanks,
>> > > > > > > > >>>>>> John
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry Shafran
- NOAA
>> > > > > > > > >>>>>> Affiliate
>> > > > > > > > >>>>>> <
>> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>
>> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
output
>> file,
>> > > but
>> > > > > > > > >>>>>>> it's
>> > > > > > > > >>>>>>> empty. Here's the output from point_stat:
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> DEBUG 4:
>> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
>> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
>> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
>> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
>> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
>> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
>> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
>> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus COPO/A1,
for
>> > > > observation
>> > > > > > > > >>>>>>> type
>> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
>> interpolation
>> > > > method
>> > > > > > > > >>>>>>> NEAREST(1),
>> > > > > > > > >>>>>>> using 0 pairs.
>> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
>> > > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
>> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Looks like most rejections were due to valid
time.
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Have any thoughts about this one?
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Thanks!
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> Perry
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry
Shafran -
>> NOAA
>> > > > > > > > >>>>>>> Affiliate
>> > > > > > > > >>>>>>> <
>> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>>
>> > > > > > > > >>>>>>>> Hi, John,
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> Looks like the changing of the time stamp
from _04
>> to
>> > > _05
>> > > > > > > > >>>>>>>> did
>> > > > > > > > >>>>>>>> the
>> > > > > > > > >>>>>>>> trick as point_stat ran without an issue
with that
>> > > change.
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> Perry
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John
Halley
>> Gotway <
>> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>
>> > > > > > > > >>>>>>>>> Perry,
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
requesting
>> > the
>> > > > > > > > >>>>>>>>> timestamp
>> > > > > > > > >>>>>>>>> of the data from that file. I think
switching the
>> > > > > > > > >>>>>>>>> verification time to
>> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix
it.
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on the
>> > > > initialization
>> > > > > > > > >>>>>>>>> times
>> > > > > > > > >>>>>>>>> and valid times for this data. Here's some
useful
>> > > wgrib2
>> > > > > > > > >>>>>>>>> and
>> > > > > > > > >>>>>>>>> MET commands
>> > > > > > > > >>>>>>>>> to illustrate.
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time (-
t),
>> > forecast
>> > > > > > > > >>>>>>>>> time
>> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it is
using
>> a
>> > > > sample
>> > > > > > > > >>>>>>>>> data file you
>> > > > > > > > >>>>>>>>> sent to me. But remember I'm still using
the
>> "old"
>> > > > wgrib2
>> > > > > > > > >>>>>>>>> so
>> > > > > > > > >>>>>>>>> I have that
>> > > > > > > > >>>>>>>>> funky forecast hour:
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
>> > > > > > > > >>>>>>>>> aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624
hour ave
>> > > > > > > > >>>>>>>>> fcst:vt=2018070505
>> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
>> > fcst:vt=2018070605
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a
higher
>> > verbosity
>> > > > > > > > >>>>>>>>> level
>> > > > > > > > >>>>>>>>> (-v
>> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info of
the
>> data
>> > it
>> > > is
>> > > > > > > > >>>>>>>>> reading:
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> # Read the first record
>> > > > > > > > >>>>>>>>> plot_data_plane
>> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
>> > > > > > > > >>>>>>>>> ...
>> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
>> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
>> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
>> > > > > > > > >>>>>>>>> ...
>> > > > > > > > >>>>>>>>> # Read the second record
>> > > > > > > > >>>>>>>>> plot_data_plane
>> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
>> > > > > > > > >>>>>>>>> ...
>> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
>> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
>> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> Hope that helps.
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>> John
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
Shafran -
>> NOAA
>> > > > > > > > >>>>>>>>> Affiliate <
>> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>
>> > > > > > > > >>>>>>>>>> Hi, John,
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> When I implemented all the changes (and
also set
>> it
>> > up
>> > > > so
>> > > > > > > > >>>>>>>>>> it's
>> > > > > > > > >>>>>>>>>> reading the beta6 version), I used this
command:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
>> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1 prepda.2018070912.nc
>> > > > > > > > >>>>>>>>>>
>> > > > > >
>> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
>> > > > > > > > >>>>>>>>>> -v 4
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> But then I got this error:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> DEBUG 2:
>> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> WARNING:
MetGrib2DataFile::data_plane_array() -
>> No
>> > > > > > > > >>>>>>>>>> matching
>> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() -> no
fields
>> > > > matching
>> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
>> > > > > > > > >>>>>>>>>> WARNING:
>> > > > > > > > >>>>>>>>>> ERROR :
>> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() -> no
>> requested
>> > > > > > > > >>>>>>>>>> forecast
>> > > > > > > > >>>>>>>>>> data
>> > > > > > > > >>>>>>>>>> found! Exiting...
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the
wgrib2
>> output
>> > for
>> > > > the
>> > > > > > > > >>>>>>>>>> file
>> > > > > > > > >>>>>>>>>> in this case reads:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> 14 (t14a2)
>> > > > > > > > >>>>>>>>>>
>> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > >
>> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
>> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-7-
16 hour
>> ave
>> > > > fcst:
>> > > > > > > > >>>>>>>>>> 15 (t14a2)
>> > > > > > > > >>>>>>>>>>
>> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be 20180708_04?
You
>> had
>> > > > > > > > >>>>>>>>>> suggested
>> > > > > > > > >>>>>>>>>> that it should be the time stamp of the
2nd day,
>> but
>> > > > maybe
>> > > > > > > > >>>>>>>>>> it needs to be
>> > > > > > > > >>>>>>>>>> the time stamp of the first day?
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
>> elsewhere.
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John
Halley
>> Gotway <
>> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> Perry,
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air
quality vx we
>> > did,
>> > > > we
>> > > > > > > > >>>>>>>>>>> used
>> > > > > > > > >>>>>>>>>>> observation files named:
>> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs
and
>> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> And those files don't exist in the July
data you
>> > just
>> > > > > > > > >>>>>>>>>>> sent.
>> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names like:
>> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
>> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
>> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
>> > aqm.t12z.anowpm.pb.tm00
>> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
>> aqm.t12z.anowpm.pb.tm120
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually verifying
>> against
>> > the
>> > > > > > > > >>>>>>>>>>> BUFR
>> > > > > > > > >>>>>>>>>>> or
>> > > > > > > > >>>>>>>>>>> PREPBUFR.
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> Let me just tell you the key points for
the
>> config
>> > > file
>> > > > > > > > >>>>>>>>>>> settings...
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to
pass 2
>> > input
>> > > > > > > > >>>>>>>>>>> files
>> > > > > > > > >>>>>>>>>>> -
>> > > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
>> > > > > > > > >>>>>>>>>>> pb2nc \
>> > > > > > > > >>>>>>>>>>> hourly.20180705/aqm.t12z.prepbufr.tm00
\
>> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc \
>> > > > > > > > >>>>>>>>>>> PB2NCConfig \
>> > > > > > > > >>>>>>>>>>> -pbfile
>> hourly.20180704/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up the
>> > time_summary
>> > > > > > > > >>>>>>>>>>> like
>> > > > > > > > >>>>>>>>>>> this:
>> > > > > > > > >>>>>>>>>>> time_summary = {
>> > > > > > > > >>>>>>>>>>> flag = TRUE;
>> > > > > > > > >>>>>>>>>>> beg = "040000";
>> > > > > > > > >>>>>>>>>>> end = "040000";
>> > > > > > > > >>>>>>>>>>> step = 24*3600;
>> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end =
0; }
>> //
>> > > > > > > > >>>>>>>>>>> previous
>> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
>> > > > > > > > >>>>>>>>>>> grib_code = [];
>> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
>> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
>> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect obs
every
>> hour
>> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
>> summaries
>> > > when
>> > > > at
>> > > > > > > > >>>>>>>>>>> least
>> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
>> > > > > > > > >>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values
at each
>> > > station
>> > > > > > > > >>>>>>>>>>> over the
>> > > > > > > > >>>>>>>>>>> defined time period. They're identified
in the
>> > > output
>> > > > by
>> > > > > > > > >>>>>>>>>>> the message type:
>> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup the
"fcst"
>> and
>> > > > "obs"
>> > > > > > > > >>>>>>>>>>> sections like this for the max of the 1-
hour
>> data.
>> > > > > > > > >>>>>>>>>>> Instead
>> > > > > > > > >>>>>>>>>>> of specifying
>> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may be
more
>> > > > convenient
>> > > > > > > > >>>>>>>>>>> to specify the
>> > > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify
>> the
>> > > > config
>> > > > > > > > >>>>>>>>>>> file, I'd suggest
>> > > > > > > > >>>>>>>>>>> using environment variables:
>> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> fcst = {
>> > > > > > > > >>>>>>>>>>> field = [
>> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
>> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
>> > > > > > > > >>>>>>>>>>> ];
>> > > > > > > > >>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>> obs = {
>> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
>> > > > > > > > >>>>>>>>>>> field = [
>> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
>> message_type =
>> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
>> > > > > > > > >>>>>>>>>>> ];
>> > > > > > > > >>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
Halley
>> Gotway
>> > <
>> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
Shafran -
>> > NOAA
>> > > > > > > > >>>>>>>>>>>> Affiliate <
>> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all
the bc
>> grids
>> > > and
>> > > > > > > > >>>>>>>>>>>>> focus on
>> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time
to copy
>> > files
>> > > > > > > > >>>>>>>>>>>>> over
>> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
>> > > > > > > > >>>>>>>>>>>>> let him work on the files he has for
now and
>> if
>> > he
>> > > > runs
>> > > > > > > > >>>>>>>>>>>>> into problems, I'll
>> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
least the
>> > 24-hr
>> > > > > > > > >>>>>>>>>>>>> max/avg files).
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
jeff.mcqueen <
>> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
>> grids...not
>> > > for
>> > > > > > > > >>>>>>>>>>>>>> bias
>> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus grids,
To be
>> > > safe,
>> > > > > > > > >>>>>>>>>>>>>> you
>> > > > > > > > >>>>>>>>>>>>>> may want to provide
>> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran -
NOAA
>> > > Affiliate
>> > > > > > > > >>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over
to Theia
>> > for
>> > > > John
>> > > > > > > > >>>>>>>>>>>>>> to do
>> > > > > > > > >>>>>>>>>>>>>> some testing some production files.
Do you
>> > think
>> > > I
>> > > > > > > > >>>>>>>>>>>>>> should use parallels
>> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops files
are OK
>> for
>> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
>> > > > > > > > >>>>>>>>>>>>>> for AQ?
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think
that the
>> ops
>> > > > files
>> > > > > > > > >>>>>>>>>>>>>> should
>> > > > > > > > >>>>>>>>>>>>>> be OK for now.
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
jeff.mcqueen
>> <
>> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about
the
>> > > parallels.
>> > > > > > > > >>>>>>>>>>>>>>> We
>> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily
max hour
>> > > labels
>> > > > > > > > >>>>>>>>>>>>>>> but
>> > > > > > > > >>>>>>>>>>>>>>> management did not
>> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19
CMAQ
>> > upgrade.
>> > > > > > > > >>>>>>>>>>>>>>> Jeff
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran -
NOAA
>> > > > Affiliate
>> > > > > > > > >>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels have
the
>> > correct
>> > > > time
>> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if I
send
>> John
>> > > > over
>> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
>> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has
any
>> other
>> > > > > > > > >>>>>>>>>>>>>>> versions
>> > > > > > > > >>>>>>>>>>>>>>> of
>> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already have
- but
>> > they
>> > > > can
>> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
>> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new
>> does
>> > > over
>> > > > > > > > >>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
>> jeff.mcqueen <
>> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new, you
may
>> want
>> > to
>> > > > use
>> > > > > > > > >>>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output (para5).
>> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
>> > > Jianping
>> > > > > > > > >>>>>>>>>>>>>>>> also
>> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it
but
>> > > management
>> > > > > > > > >>>>>>>>>>>>>>>> did
>> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
>> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to do
so
>> with
>> > the
>> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
>> > > > > > > > >>>>>>>>>>>>>>>> implementation.
>> > > > > > > > >>>>>>>>>>>>>>>> Jeff
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran -
NOAA
>> > > > Affiliate
>> > > > > > > > >>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
>> conversation.
>> > I
>> > > > > > > > >>>>>>>>>>>>>>>> think
>> > > > > > > > >>>>>>>>>>>>>>>> I
>> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the
timestamps some
>> > time
>> > > > ago,
>> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
>> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or with
>> Jeff. I
>> > > > think
>> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
>> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not
too
>> > entirely
>> > > > > > > > >>>>>>>>>>>>>>>> sure.
>> > > > > > > > >>>>>>>>>>>>>>>> But he would have
>> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct
and how
>> to
>> > > > > > > > >>>>>>>>>>>>>>>> approach
>> > > > > > > > >>>>>>>>>>>>>>>> this.
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time
stamp
>> issues
>> > > John
>> > > > > > > > >>>>>>>>>>>>>>>> talks
>> > > > > > > > >>>>>>>>>>>>>>>> about?
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> Perry
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM, John
Halley
>> > > > Gotway <
>> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all
squared away
>> to
>> > > > > > > > >>>>>>>>>>>>>>>>> compute
>> > > > > > > > >>>>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average values
for 1-
>> and
>> > > 8-
>> > > > > > > > >>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
>> > \
>> > > > > > > > >>>>>>>>>>>>>>>>>
aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
>> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
>> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
>> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like
this:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
>> > > > > \
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
>> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
PointStatConfig_AIRNOW \
>> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
>> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1
versus
>> COPO/A1,
>> > > for
>> > > > > > > > >>>>>>>>>>>>>>>>> observation type AIRNOW_MAX_240000,
over
>> > region
>> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
>> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
>> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147
pairs.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
>> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes the
>> contents
>> > of
>> > > > > > > > >>>>>>>>>>>>>>>>> this
>> > > > > > > > >>>>>>>>>>>>>>>>> "max"
>> > > > > > > > >>>>>>>>>>>>>>>>> file:
>> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
>> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
>> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618 hour
ave
>> fcst:
>> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
sigma
>> > > > level:17-40
>> > > > > > > > >>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty funny.
But
>> > > testing
>> > > > > > > > >>>>>>>>>>>>>>>>> revealed
>> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
forecast
>> > > hours
>> > > > 17
>> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
>> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
>> > > better.
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours
are in
>> the
>> > > same
>> > > > > > > > >>>>>>>>>>>>>>>>> file, we
>> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to
tell it
>> which
>> > > one
>> > > > > > > > >>>>>>>>>>>>>>>>> to
>> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
>> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
>> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
>> > > > > > > > >>>>>>>>>>>>>>>>> field = [
>> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level =
"L1";
>> > > > lead_time
>> > > > > > > > >>>>>>>>>>>>>>>>> =
>> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
>> > > > > > > > >>>>>>>>>>>>>>>>> ];
>> > > > > > > > >>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
>> timestamps
>> > are
>> > > > off
>> > > > > > > > >>>>>>>>>>>>>>>>> by 1
>> > > > > > > > >>>>>>>>>>>>>>>>> hour.
>> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus a
41 hour
>> > lead
>> > > > > > > > >>>>>>>>>>>>>>>>> time
>> > > > > > > > >>>>>>>>>>>>>>>>> =
>> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
>> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a
24-hour
>> max
>> > > > value
>> > > > > > > > >>>>>>>>>>>>>>>>> from
>> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
valid at
>> 05Z
>> > > > while
>> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
>> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
>> Point-Stat
>> > > > config
>> > > > > > > > >>>>>>>>>>>>>>>>> file by
>> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look
back 1
>> hour:
>> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
>> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
>> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
>> > > > > > > > >>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM
hsoh
>> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
>> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files
and was
>> > > fixed.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John Halley
Gotway
>> > > wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the time
>> summaries,
>> > > > but I
>> > > > > > > > >>>>>>>>>>>>>>>>>> found
>> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a
look on
>> > dakota
>> > > > > > > > >>>>>>>>>>>>>>>>>> here:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > /d3/projects/MET/MET_Help/shafran_data_20180626
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of pb2nc
to
>> compute
>> > > 24-
>> > > > > > > > >>>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped
>> the
>> > > > output
>> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
>> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands
>> to
>> > > count
>> > > > > > > > >>>>>>>>>>>>>>>>>> up
>> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
>> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
>> > > pb2nc_output.txt
>> > > > |
>> > > > > > > > >>>>>>>>>>>>>>>>>> wc
>> > > > > > > > >>>>>>>>>>>>>>>>>> -l
>> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
>> > > pb2nc_output.txt
>> > > > |
>> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
>> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
>> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
>> > pb2nc_output.txt |
>> > > > wc
>> > > > > > > > >>>>>>>>>>>>>>>>>> -l
>> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
>> > pb2nc_output.txt |
>> > > > > > > > >>>>>>>>>>>>>>>>>> sort
>> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
>> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return 114,538...
>> meaning
>> > > that
>> > > > > > > > >>>>>>>>>>>>>>>>>> all
>> > > > > > > > >>>>>>>>>>>>>>>>>> of
>> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
unique.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and (4)
returns
>> > > 6,555
>> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
>> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations
>> are
>> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID
in that
>> > file
>> > > is
>> > > > > > > > >>>>>>>>>>>>>>>>>> named
>> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
pb2nc_output.txt |
>> > grep
>> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180527_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180527_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
>> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
>> 20180528_040000
>> > > 32.47
>> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
>> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
>> repeated in
>> > > the
>> > > > > > > > >>>>>>>>>>>>>>>>>> last two
>> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see why
these
>> > > > duplicates
>> > > > > > > > >>>>>>>>>>>>>>>>>> are
>> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM
hsoh
>> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
>> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John Halley
Gotway
>> > > wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's
no
>> reason
>> > why
>> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
>> > > > > > > > >>>>>>>>>>>>>>>>>>> have
>> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
>> > > across 2
>> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
>> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from
which we
>> > could
>> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
>> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard
>> Soh
>> > <
>> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about 10
minutes
>> > > > (550.319
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
minutes.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will
not
>> check
>> > in
>> > > > the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
files.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02 PM
hsoh
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the output
(the
>> > content
>> > > > of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test for
time
>> > > summary
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple
BURF
>> > inputs.
>> > > I
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two input
files
>> and
>> > > > will
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
Halley
>> Gotway
>> > > > wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this
will
>> change
>> > > the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this case
not
>> > > currently
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I add
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00 PM
Howard
>> > Soh
>> > > <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset for
time
>> > > summary
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to
SVN.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
>> /d3/personal/hsoh/data/pb2nc_summary/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52
PM John
>> > > Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates
to the
>> > logic
>> > > > for
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying to
>> compute
>> > > > 24-hour
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see these
obs/hdr
>> > > > counts:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; // (121832
>> > currently)
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
Rscript
>> to
>> > > dump
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
AIRNOW_MAX_240000
>> > > string
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem
is in
>> PB2NC
>> > > not
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
summaries... or
>> in
>> > the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take another
look in
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:54
PM John
>> > > Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging into
this.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at 1:41
PM
>> Howard
>> > > > Soh <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291
observations from
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> aqm.20180527/aqm.t12z.prepbufr.tm00
>> > > and
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
>> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem for
the obs
>> > > data.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at
2:28 PM
>> John
>> > > > Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
Could you
>> > > please
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
>> > PB2NC.
>> > > > I
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc
PB2NCConfig \
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
request
>> > > 24-hourly
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z,
like
>> this:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
-24*3600;
>> > end =
>> > > > 0;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO" ];
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max" ];
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
observations...
>> > but
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which is
not
>> what
>> > > you'd
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The hdr_typ_table
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
>> > > > "AIRNOW_MAX_240000"
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
pointed
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
referenced.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
>> > caused
>> > > by
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I
need 2
>> of
>> > > them
>> > > > to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to
04Z
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a
look?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:44 PM
>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
>> > > PrepBUFR
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting to
>> compute a
>> > 24
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more together
next
>> > week.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:40 PM
>> > hsoh
>> > > <
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R script
is
>> > > reasonable
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
observation
>> > data
>> > > in
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
index of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the NetCDF
file
>> was
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from
pb2nc
>> > without
>> > > > any
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it has
MORE
>> > > headers
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
118193=headers).
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables from
the
>> > NetCDF
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(7295
>> > > currently)
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
value
>> from 0
>> > > to
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message
type for
>> > > > summary)
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of hdr_typ
>> variable.
>> > > The
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger
>> than
>> > > > 3655.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems to
be OK.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > >
>> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MEAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEDIAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_RANGE_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_SDEV_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MEAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEDIAN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_RANGE_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_SDEV_010000
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM,
John
>> Halley
>> > > > Gotway
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a
look on
>> > dakota
>> > > > in
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to
dump it
>> to
>> > > > ascii:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
>> > > > /d1/johnhg/MET/MET_development/
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
>> > pb2nc_output.txt
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error
but lists
>> > all
>> > > of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as AIRNOW.
But
>> some
>> > > of
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000... as
>> evidenced
>> > by
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v hdr_typ_table
>> > > > pb2nc_output.nc
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" ;
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
indexing or
>> > table
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
>> > > > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Fri Jun 07 05:45:06 2019
Hi, John,
Thanks!
Perry
On Thu, Jun 6, 2019 at 6:21 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Perry (and Howard),
>
> I isolated this issue down and wrote up a GitHub issue describing
the
> problem:
> https://github.com/NCAR/MET/issues/1137
>
> Howard, please take a look and let me know if you need any help
further
> debugging this.
>
> Thanks,
> John
>
> On Thu, Jun 6, 2019 at 3:21 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Perry,
> >
> > I was able to replicate this behavior both on WCOSS and locally.
I
> > believe it's a bug in PB2NC. I'm trying to narrow it down and
will then
> > hand it off to Howard to debug. Thanks for identifying this
issue.
> >
> > John
> >
> > On Wed, Jun 5, 2019 at 3:07 PM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >>
> >> Hi, John,
> >>
> >> Well, it does appear that I got the same error with MET 8.1 as I
did
> with
> >> the previous version:
> >>
> >> ERROR :
> >> ERROR : StringArray::operator[](int) const -> range check error!
> >> ERROR :
> >>
> >> Perry
> >>
> >> On Mon, Jun 3, 2019 at 2:16 PM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Perry,
> >> >
> >> > Did you ever get this issue resolved from 1 month ago? Or
should I
> log
> >> on
> >> > to WCOSS and take a look?
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Fri, May 3, 2019 at 10:06 AM perry.shafran at noaa.gov via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Hi, John,
> >> > >
> >> > > Well, I am running the script in batch so I don't control
what node
> it
> >> > > gets. I can tell you that the working directory is here:
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > >
> >> > > And the output log file is here:
> >> > >
/ptmpp2/Perry.Shafran/output/verf_gridtobs_aqmmax_12.o1502688, which
> >> > should
> >> > > help you figure out how the thing is run.
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Perry
> >> > >
> >> > > On Fri, May 3, 2019 at 12:02 PM John Halley Gotway via RT <
> >> > > met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Perry,
> >> > > >
> >> > > > We can log on to wcoss and test there instead. Yes, pb2nc
does
> >> seem to
> >> > > be
> >> > > > the culprit. Can you point us the wcoss node you're using
and the
> >> > > working
> >> > > > directory there? We basically need to recreate your call
to pb2nc
> >> to
> >> > try
> >> > > > to replicate the behavior.
> >> > > >
> >> > > > Thanks,
> >> > > > John
> >> > > >
> >> > > > On Fri, May 3, 2019 at 6:36 AM perry.shafran at noaa.gov via
RT <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89921 >
> >> > > > >
> >> > > > > Hi, David, John,
> >> > > > >
> >> > > > > I am not running on Theia. In fact I think it would take
quite
> a
> >> bit
> >> > > of
> >> > > > > effort to transition this thing to Theia, and I'll need
to be
> >> running
> >> > > > this
> >> > > > > verification on prod data anyway, which is only on WCOSS.
> >> > > > >
> >> > > > > However, if you wish to troubleshoot on Theia, you should
do so,
> >> and
> >> > I
> >> > > > > think you have all you need to be able to do so, right?
> >> > > > >
> >> > > > > It would seem that the error on my side is in PB2NC, so
I'll
> >> probably
> >> > > > need
> >> > > > > to make sure I'm running what you're running.
> >> > > > >
> >> > > > > Thanks!
> >> > > > >
> >> > > > > Perry
> >> > > > >
> >> > > > > On Thu, May 2, 2019 at 5:43 PM David Fillmore via RT <
> >> > > met_help at ucar.edu>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Perry -
> >> > > > > > Are you running on theia?
> >> > > > > > John suggests that we try to troubleshoot there ...
> >> > > > > > thanks,
> >> > > > > > David
> >> > > > > >
> >> > > > > > On Thu May 02 15:35:10 2019, fillmore wrote:
> >> > > > > > > Hi Perry -
> >> > > > > > >
> >> > > > > > > An update on where we are on this issue:
> >> > > > > > > Using the prepda.2019042312.nc file you provided we
indeed
> >> > > reproduce
> >> > > > > > > the point_stat error.
> >> > > > > > > The point_stat error is due to only a single entry
the valid
> >> time
> >> > > > > > > table,
> >> > > > > > > e.g. from ncdump -v hdr_vld_table
> >> > > > > > > hdr_vld_table =
> >> > > > > > > "20190423_050000",
> >> > > > > > > while the hdr_vld indices in that file are hdr_vld =
0, 33,
> >> 33,
> >> > ...
> >> > > > > > > which are all out of range except the first.
> >> > > > > > >
> >> > > > > > > If I run pb2nc here with your PREPBUFR data and your
> >> > > PB2NCConfig_aqm
> >> > > > > > > config file, however,
> >> > > > > > > I generate a different prepda.2019042312.nc file that
runs
> >> > through
> >> > > > to
> >> > > > > > > completion with point_stat.
> >> > > > > > > Its valid time table:
> >> > > > > > >
> >> > > > > > > hdr_vld_table =
> >> > > > > > > "20190423_030000",
> >> > > > > > > "20190423_010000",
> >> > > > > > > "20190423_020000",
> >> > > > > > > "20190423_230000",
> >> > > > > > > "20190423_150000",
> >> > > > > > > "20190423_090000",
> >> > > > > > > "20190423_070000",
> >> > > > > > > "20190423_060000",
> >> > > > > > > "20190423_210000",
> >> > > > > > > "20190423_050000",
> >> > > > > > > "20190423_170000",
> >> > > > > > > "20190423_100000",
> >> > > > > > > "20190423_200000",
> >> > > > > > > "20190423_140000",
> >> > > > > > > "20190423_130000",
> >> > > > > > > "20190423_160000",
> >> > > > > > > "20190423_040000",
> >> > > > > > > "20190423_180000",
> >> > > > > > > "20190423_080000",
> >> > > > > > > "20190423_120000",
> >> > > > > > > "20190423_110000",
> >> > > > > > > "20190423_220000",
> >> > > > > > > "20190423_190000",
> >> > > > > > > "20190424_000000" ;
> >> > > > > > > }
> >> > > > > > >
> >> > > > > > > So it seems pb2nc is running correctly here, but not
on your
> >> > side.
> >> > > > > > > Let me discuss with John the best way to proceed ...
> >> > > > > > > thanks,
> >> > > > > > > David
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Thu Apr 25 11:16:42 2019, perry.shafran at noaa.gov
wrote:
> >> > > > > > > > Hi, everyone,
> >> > > > > > > >
> >> > > > > > > > I am attaching this to this long thread regarding
the
> daily
> >> max
> >> > > of
> >> > > > > > > > ozone
> >> > > > > > > > verification, so we remember what we were doing
back then.
> >> > > > > > > >
> >> > > > > > > > I am now running this again using MET 8.0, and
something
> >> seems
> >> > to
> >> > > > not
> >> > > > > > > > be
> >> > > > > > > > working when I run point_stat. I get this error
when
> >> running a
> >> > > > daily
> >> > > > > > > > max
> >> > > > > > > > of ozone:
> >> > > > > > > >
> >> > > > > > > > ERROR :
> >> > > > > > > > ERROR : StringArray::operator[](int) const ->
range check
> >> > error!
> >> > > > > > > > ERROR :
> >> > > > > > > >
> >> > > > > > > > I am using my own scripts to run this using MET, so
this
> is
> >> not
> >> > > > using
> >> > > > > > > > METplus.
> >> > > > > > > >
> >> > > > > > > > You can find a log file of this here:
> >> > > > > > > >
> >> > > > >
> >> >
>
/meso/noscrub/Perry.Shafran/aqm_issue/verf_gridtobs_aqmmax_12.o1210430.
> >> > > > > > > > It
> >> > > > > > > > runs pb2nc and point_stat for max ozone. The pb2nc
code is
> >> > > working
> >> > > > > > > > fine, I
> >> > > > > > > > believe.
> >> > > > > > > >
> >> > > > > > > > The script I am using to run this is here:
> >> > > > > > > >
> >> >
/meso/save/Perry.Shafran/verif/nwtest/ush/verf_gridtobs_aqmmax.sh
> >> > > > > > > >
> >> > > > > > > > The prepbufr files are:
> >> > > > > > > >
> >> > > > > > > >
/com/hourly/prod/hourly.20190422/aqm.t12z.prepbufr.tm00
> and
> >> > > > > > > >
/com/hourly/prod/hourly.20190423/aqm.t12z.prepbufr.tm00
> >> > > > > > > >
> >> > > > > > > > The model file is here:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/meso/noscrub/Perry.Shafran/aqm_issue/tmpnwtest/verf_gridtobs_aqmmax_12/oz_DAY1
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > What could be causing this error?
> >> > > > > > > >
> >> > > > > > > > Thanks!
> >> > > > > > > >
> >> > > > > > > > Perry
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Wed, Aug 15, 2018 at 4:12 PM John Halley Gotway
<
> >> > > > johnhg at ucar.edu>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Great, glad that did the trick.
> >> > > > > > > > >
> >> > > > > > > > > John
> >> > > > > > > > >
> >> > > > > > > > > On Wed, Aug 15, 2018 at 1:39 PM Perry Shafran -
NOAA
> >> > Affiliate
> >> > > <
> >> > > > > > > > > perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Well this worked! And Jeff confirmed to me that
the
> 8-hr
> >> > > > average
> >> > > > > > > > >> actually is a 12Z to 12Z average, not 5Z to 5Z
like in
> >> the
> >> > > 1-hr
> >> > > > > > > > >> avg.
> >> > > > > > > > >>
> >> > > > > > > > >> Thanks!
> >> > > > > > > > >>
> >> > > > > > > > >> Perry
> >> > > > > > > > >>
> >> > > > > > > > >> On Wed, Aug 15, 2018 at 1:43 PM, John Halley
Gotway
> >> > > > > > > > >> <johnhg at ucar.edu>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >>
> >> > > > > > > > >>> Perry,
> >> > > > > > > > >>>
> >> > > > > > > > >>> I think the problem is in your setting of
> >> > > > > > > > >>> valid_time="${CUR_VX_TIME}";
> >> > > > > > > > >>>
> >> > > > > > > > >>> Here's your wgrib2 output:
> >> > > > > > > > >>> wgrib2 oz_DAY1
> >> > > > > > > > >>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour
> ave
> >> > fcst:
> >> > > > > > > > >>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-
1-22
> hour
> >> > ave
> >> > > > > > > > >>> fcst:
> >> > > > > > > > >>>
> >> > > > > > > > >>> The "d=2018081112" tells you the model
initialization
> >> time.
> >> > > > The
> >> > > > > > > > >>> valid
> >> > > > > > > > >>> time = init time + forecast time. So the
OZMAX1 data
> is
> >> > > valid
> >> > > > 16
> >> > > > > > > > >>> hours
> >> > > > > > > > >>> later while the OZMAX8 data is valid 22 hours
later.
> >> Rerun
> >> > > > that
> >> > > > > > > > >>> wgrib2
> >> > > > > > > > >>> command using the "-vt" argument to list the
> >> verification
> >> > > time.
> >> > > > > > > > >>>
> >> > > > > > > > >>> Point-Stat isn't finding any OZMAX8 data
because that
> >> data
> >> > > > isn't
> >> > > > > > > > >>> valid
> >> > > > > > > > >>> at the time you're requesting.
> >> > > > > > > > >>>
> >> > > > > > > > >>> John
> >> > > > > > > > >>>
> >> > > > > > > > >>>
> >> > > > > > > > >>> On Wed, Aug 15, 2018 at 9:14 AM Perry Shafran -
NOAA
> >> > > Affiliate
> >> > > > <
> >> > > > > > > > >>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>
> >> > > > > > > > >>>> Hi, John,
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I haven't been able to work on this for
awhile, but
> now
> >> > I'm
> >> > > > > > > > >>>> working on
> >> > > > > > > > >>>> it again but seem to be running into issues.
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I have a file that has both the daily max for
1-hr
> >> ozone
> >> > and
> >> > > > the
> >> > > > > > > > >>>> daily
> >> > > > > > > > >>>> max for 8-hr ozone. The wgrib2 output looks
like
> this:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> 9 (t10a2)
> >> > > > > > > > >>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>> >
> >> > > > > > > > >>>> wgrib2 oz_DAY1
> >> > > > > > > > >>>> 1:0:d=2018081112:OZMAX1:1 sigma level:-7-16
hour ave
> >> fcst:
> >> > > > > > > > >>>> 2:148490:d=2018081112:OZMAX8:1 sigma level:-1-
22 hour
> >> ave
> >> > > > fcst:
> >> > > > > > > > >>>> 10 (t10a2)
> >> > > > > > > > >>>>
> /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>> >
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> As you can see they have different forecast
times.
> But
> >> > > when I
> >> > > > > > > > >>>> run
> >> > > > > > > > >>>> point_stat, I get this output:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>>
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------
> >> > > > > > > > >>>> DEBUG 2:
> >> > > > > > > > >>>> DEBUG 2: Reading data for OZMAX8/L1.
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING: MetGrib2DataFile::data_plane_array()
- No
> >> > matching
> >> > > > > > > > >>>> records
> >> > > > > > > > >>>> found for 'OZMAX8/L1'
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>> WARNING: process_fcst_climo_files() -> no
fields
> >> matching
> >> > > > > > > > >>>> OZMAX8/L1
> >> > > > > > > > >>>> found in file: oz_DAY1
> >> > > > > > > > >>>> WARNING:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Here is the relevant portion of the point_stat
config
> >> > file:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> fcst = {
> >> > > > > > > > >>>> field = [
> >> > > > > > > > >>>> { name = "OZMAX1"; level = "L1";
> >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; },
> >> > > > > > > > >>>> { name = "OZMAX8"; level = "L1";
> >> > > > > > > > >>>> valid_time="${CUR_VX_TIME}"; }
> >> > > > > > > > >>>> ];
> >> > > > > > > > >>>> }
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> obs = {
> >> > > > > > > > >>>> convert(x) = x * 10^9;
> >> > > > > > > > >>>> field = [
> >> > > > > > > > >>>> { name = "COPO"; level = "A1";
message_type =
> >> > > > > > > > >>>> "AIRNOW_MAX_230000"; },
> >> > > > > > > > >>>> { name = "COPO"; level = "A8";
message_type =
> >> > > > > > > > >>>> "AIRNOW_MAX_230000"; }
> >> > > > > > > > >>>> ];
> >> > > > > > > > >>>> }
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> I don't remember if I had this working
previously for
> >> both
> >> > > the
> >> > > > > > > > >>>> 1hr
> >> > > > > > > > >>>> and
> >> > > > > > > > >>>> the 8hr ozone, but apparently there is some
sort of
> >> issue
> >> > > for
> >> > > > > > > > >>>> the
> >> > > > > > > > >>>> 8hr
> >> > > > > > > > >>>> ozone. Have any thoughts?
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Thanks!
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> Perry
> >> > > > > > > > >>>>
> >> > > > > > > > >>>> On Wed, Jul 11, 2018 at 2:44 PM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>> <
> >> > > > > > > > >>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>
> >> > > > > > > > >>>>> I chose option 2, and that seems to work - I
have a
> >> .stat
> >> > > > file
> >> > > > > > > > >>>>> with
> >> > > > > > > > >>>>> output. Now I'll need to do some work here
to see
> if
> >> the
> >> > > > > > > > >>>>> output
> >> > > > > > > > >>>>> looks OK,
> >> > > > > > > > >>>>> and to configure the verification so it
verifies
> >> OZMAX1
> >> > and
> >> > > > > > > > >>>>> OZMAX8.
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> I also wonder if this would be easily
modified for
> PM.
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> Perry
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>> On Wed, Jul 11, 2018 at 2:11 PM, John Halley
Gotway
> >> > > > > > > > >>>>> <johnhg at ucar.edu>
> >> > > > > > > > >>>>> wrote:
> >> > > > > > > > >>>>>
> >> > > > > > > > >>>>>> Perry,
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> Yep, you've configured/run PB2NC to compute
time
> >> > summaries
> >> > > > > > > > >>>>>> that
> >> > > > > > > > >>>>>> are
> >> > > > > > > > >>>>>> valid at 04Z. And you're comparing them to
> forecasts
> >> > that
> >> > > > are
> >> > > > > > > > >>>>>> purportedly
> >> > > > > > > > >>>>>> valid at 05Z. And that's the reason why you
have
> so
> >> > many
> >> > > > obs
> >> > > > > > > > >>>>>> rejected due
> >> > > > > > > > >>>>>> to the valid time.
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> It's up to you guys to decide what the
"correct"
> way
> >> of
> >> > > > > > > > >>>>>> handling
> >> > > > > > > > >>>>>> this
> >> > > > > > > > >>>>>> is... but here are a couple of options:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> (1) You could use the existing setup and
address
> this
> >> > > logic
> >> > > > in
> >> > > > > > > > >>>>>> Point-Stat config file. Change the matching
time
> >> > > obs_window
> >> > > > > > > > >>>>>> from 0 to:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> obs_window = {
> >> > > > > > > > >>>>>> beg = 3600;
> >> > > > > > > > >>>>>> end = 3600;
> >> > > > > > > > >>>>>> }
> >> > > > > > > > >>>>>> This tells Point-Stat to match observations
which
> are
> >> > > > exactly
> >> > > > > > > > >>>>>> 1-
> >> > > > > > > > >>>>>> hour
> >> > > > > > > > >>>>>> prior to the valid time of the forecast.
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> (2) You could rerun PB2NC to compute a time
summary
> >> from
> >> > > 05Z
> >> > > > > > > > >>>>>> to
> >> > > > > > > > >>>>>> 05Z
> >> > > > > > > > >>>>>> instead and then leave the Point-Stat
obs_window
> >> setting
> >> > > at
> >> > > > 0:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> time_summary = {
> >> > > > > > > > >>>>>> flag = TRUE;
> >> > > > > > > > >>>>>> beg = "050000";
> >> > > > > > > > >>>>>> end = "050000";
> >> > > > > > > > >>>>>> ...
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> Thanks,
> >> > > > > > > > >>>>>> John
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>> On Wed, Jul 11, 2018 at 12:02 PM Perry
Shafran -
> NOAA
> >> > > > > > > > >>>>>> Affiliate
> >> > > > > > > > >>>>>> <
> >> > > > > > > > >>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>
> >> > > > > > > > >>>>>>> Oops, spoke too soon. There *is* a .stat
output
> >> file,
> >> > > but
> >> > > > > > > > >>>>>>> it's
> >> > > > > > > > >>>>>>> empty. Here's the output from point_stat:
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> DEBUG 4:
> >> > > > > > > > >>>>>>> DEBUG 4: Data plane information:
> >> > > > > > > > >>>>>>> DEBUG 4: plane min: 13.86
> >> > > > > > > > >>>>>>> DEBUG 4: plane max: 130.82
> >> > > > > > > > >>>>>>> DEBUG 4: scan mode: 64
> >> > > > > > > > >>>>>>> DEBUG 4: valid time: 20180709_050000
> >> > > > > > > > >>>>>>> DEBUG 4: lead time: 170000
> >> > > > > > > > >>>>>>> DEBUG 4: init time: 20180708_120000
> >> > > > > > > > >>>>>>> DEBUG 4: bitmap flag: 255
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> DEBUG 2: Processing OZMAX1/L1 versus
COPO/A1, for
> >> > > > observation
> >> > > > > > > > >>>>>>> type
> >> > > > > > > > >>>>>>> AIRNOW_MAX_230000, over region FULL, for
> >> interpolation
> >> > > > method
> >> > > > > > > > >>>>>>> NEAREST(1),
> >> > > > > > > > >>>>>>> using 0 pairs.
> >> > > > > > > > >>>>>>> DEBUG 3: Number of matched pairs = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Observations processed = 124637
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: SID exclusion = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: obs type = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: valid time = 122154
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad obs value = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: off the grid = 70
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: level mismatch = 1224
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: quality marker = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: message type = 1189
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: masking region = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: bad fcst value = 0
> >> > > > > > > > >>>>>>> DEBUG 3: Rejected: duplicates = 0
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Looks like most rejections were due to
valid time.
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Have any thoughts about this one?
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Thanks!
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> Perry
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>> On Wed, Jul 11, 2018 at 1:13 PM, Perry
Shafran -
> >> NOAA
> >> > > > > > > > >>>>>>> Affiliate
> >> > > > > > > > >>>>>>> <
> >> > > > > > > > >>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>
> >> > > > > > > > >>>>>>>> Hi, John,
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Looks like the changing of the time stamp
from
> _04
> >> to
> >> > > _05
> >> > > > > > > > >>>>>>>> did
> >> > > > > > > > >>>>>>>> the
> >> > > > > > > > >>>>>>>> trick as point_stat ran without an issue
with
> that
> >> > > change.
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> Perry
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>> On Wed, Jul 11, 2018 at 12:12 PM, John
Halley
> >> Gotway <
> >> > > > > > > > >>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>
> >> > > > > > > > >>>>>>>>> Perry,
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> I agree, there's likely some issue with
> requesting
> >> > the
> >> > > > > > > > >>>>>>>>> timestamp
> >> > > > > > > > >>>>>>>>> of the data from that file. I think
switching
> the
> >> > > > > > > > >>>>>>>>> verification time to
> >> > > > > > > > >>>>>>>>> from 20180709_04 to 20180709_05 would fix
it.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Looks like wgrib2 and MET both agree on
the
> >> > > > initialization
> >> > > > > > > > >>>>>>>>> times
> >> > > > > > > > >>>>>>>>> and valid times for this data. Here's
some
> useful
> >> > > wgrib2
> >> > > > > > > > >>>>>>>>> and
> >> > > > > > > > >>>>>>>>> MET commands
> >> > > > > > > > >>>>>>>>> to illustrate.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> (1) Use wgrib2 to tell you the init time
(-t),
> >> > forecast
> >> > > > > > > > >>>>>>>>> time
> >> > > > > > > > >>>>>>>>> (-ftime), and valid time (-vt). Here it
is
> using
> >> a
> >> > > > sample
> >> > > > > > > > >>>>>>>>> data file you
> >> > > > > > > > >>>>>>>>> sent to me. But remember I'm still using
the
> >> "old"
> >> > > > wgrib2
> >> > > > > > > > >>>>>>>>> so
> >> > > > > > > > >>>>>>>>> I have that
> >> > > > > > > > >>>>>>>>> funky forecast hour:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> wgrib2 -t -ftime -vt
> >> > > > > > > > >>>>>>>>>
aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> 1:0:d=2018070406:-2147483647--2147483624
hour
> ave
> >> > > > > > > > >>>>>>>>> fcst:vt=2018070505
> >> > > > > > > > >>>>>>>>> 2:144131:d=2018070406:23-46 hour ave
> >> > fcst:vt=2018070605
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> (2) Run the plot_data_plane tool at a
higher
> >> > verbosity
> >> > > > > > > > >>>>>>>>> level
> >> > > > > > > > >>>>>>>>> (-v
> >> > > > > > > > >>>>>>>>> 4) to have MET tell you the timing info
of the
> >> data
> >> > it
> >> > > is
> >> > > > > > > > >>>>>>>>> reading:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> # Read the first record
> >> > > > > > > > >>>>>>>>> plot_data_plane
> >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> rec1.ps 'name="OZMAX1"; level="R1";' -v 4
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180705_050000
> >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 230000
> >> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> # Read the second record
> >> > > > > > > > >>>>>>>>> plot_data_plane
> >> > > > aqm.20180704/aqm.t06z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>> rec2.ps 'name="OZMAX1"; level="R2";' -v 4
> >> > > > > > > > >>>>>>>>> ...
> >> > > > > > > > >>>>>>>>> DEBUG 4: valid time: 20180706_050000
> >> > > > > > > > >>>>>>>>> DEBUG 4: lead time: 470000
> >> > > > > > > > >>>>>>>>> DEBUG 4: init time: 20180704_060000
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Hope that helps.
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>> John
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>> On Wed, Jul 11, 2018 at 8:53 AM Perry
Shafran -
> >> NOAA
> >> > > > > > > > >>>>>>>>> Affiliate <
> >> > > > > > > > >>>>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Hi, John,
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> When I implemented all the changes (and
also
> set
> >> it
> >> > up
> >> > > > so
> >> > > > > > > > >>>>>>>>>> it's
> >> > > > > > > > >>>>>>>>>> reading the beta6 version), I used this
> command:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> CUR_VX_TIME=20180709_04
> >> > > > > > > > >>>>>>>>>> point_stat oz1hr_DAY1
prepda.2018070912.nc
> >> > > > > > > > >>>>>>>>>>
> >> > > > > >
> >> >
/meso/save/Perry.Shafran/verif/nwtest/parm/PointStatConfig_AIRNOW_max
> >> > > > > > > > >>>>>>>>>> -v 4
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> But then I got this error:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> DEBUG 2:
> >> > > > > > > > >>>>>>>>>> DEBUG 2: Reading data for OZMAX1/L1.
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING:
MetGrib2DataFile::data_plane_array() -
> >> No
> >> > > > > > > > >>>>>>>>>> matching
> >> > > > > > > > >>>>>>>>>> records found for 'OZMAX1/L1'
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> WARNING: process_fcst_climo_files() ->
no
> fields
> >> > > > matching
> >> > > > > > > > >>>>>>>>>> OZMAX1/L1 found in file: oz1hr_DAY1
> >> > > > > > > > >>>>>>>>>> WARNING:
> >> > > > > > > > >>>>>>>>>> ERROR :
> >> > > > > > > > >>>>>>>>>> ERROR : process_fcst_climo_files() ->
no
> >> requested
> >> > > > > > > > >>>>>>>>>> forecast
> >> > > > > > > > >>>>>>>>>> data
> >> > > > > > > > >>>>>>>>>> found! Exiting...
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Not sure why that is. A look at the
wgrib2
> >> output
> >> > for
> >> > > > the
> >> > > > > > > > >>>>>>>>>> file
> >> > > > > > > > >>>>>>>>>> in this case reads:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> 14 (t14a2)
> >> > > > > > > > >>>>>>>>>>
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > >
> >> > > > > > > > >>>>>>>>>> wgrib2 oz1hr_DAY1
> >> > > > > > > > >>>>>>>>>> 1:0:d=2018070812:OZMAX1:1 sigma level:-
7-16
> hour
> >> ave
> >> > > > fcst:
> >> > > > > > > > >>>>>>>>>> 15 (t14a2)
> >> > > > > > > > >>>>>>>>>>
> >> > > /stmpp2/Perry.Shafran/tmpnwtest/verf_gridtobs_aqmmax_12
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Maybe CUR_VX_TIME needs to be
20180708_04? You
> >> had
> >> > > > > > > > >>>>>>>>>> suggested
> >> > > > > > > > >>>>>>>>>> that it should be the time stamp of the
2nd
> day,
> >> but
> >> > > > maybe
> >> > > > > > > > >>>>>>>>>> it needs to be
> >> > > > > > > > >>>>>>>>>> the time stamp of the first day?
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Unless I'm missing something in pb2nc or
> >> elsewhere.
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Thanks for any thoughts.
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>> On Fri, Jul 6, 2018 at 5:44 PM, John
Halley
> >> Gotway <
> >> > > > > > > > >>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> Perry,
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> I'm confused. In the previous air
quality vx
> we
> >> > did,
> >> > > > we
> >> > > > > > > > >>>>>>>>>>> used
> >> > > > > > > > >>>>>>>>>>> observation files named:
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.prepbufr.tm00 for AIRNOW obs
and
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024 for ANOWPM obs
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> And those files don't exist in the July
data
> you
> >> > just
> >> > > > > > > > >>>>>>>>>>> sent.
> >> > > > > > > > >>>>>>>>>>> Instead, it's BUFR files with names
like:
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm00.bufr_d
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm024.bufr_d
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.airnow.tm120.bufr_d
> >> > aqm.t12z.anowpm.pb.tm00
> >> > > > > > > > >>>>>>>>>>> aqm.t12z.anowpm.pb.tm024
> >> aqm.t12z.anowpm.pb.tm120
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> I'm not sure if you're actually
verifying
> >> against
> >> > the
> >> > > > > > > > >>>>>>>>>>> BUFR
> >> > > > > > > > >>>>>>>>>>> or
> >> > > > > > > > >>>>>>>>>>> PREPBUFR.
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> Let me just tell you the key points for
the
> >> config
> >> > > file
> >> > > > > > > > >>>>>>>>>>> settings...
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (1) When you call pb2nc, you'll need to
pass 2
> >> > input
> >> > > > > > > > >>>>>>>>>>> files
> >> > > > > > > > >>>>>>>>>>> -
> >> > > > > > > > >>>>>>>>>>> thecurrent day and the previous day...
> >> > > > > > > > >>>>>>>>>>> pb2nc \
> >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00 \
> >> > > > > > > > >>>>>>>>>>>
hourly.20180705/aqm.t12z.prepbufr.tm00.nc
> \
> >> > > > > > > > >>>>>>>>>>> PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>> -pbfile
> >> hourly.20180704/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (2) In the PB2NC config file, set up
the
> >> > time_summary
> >> > > > > > > > >>>>>>>>>>> like
> >> > > > > > > > >>>>>>>>>>> this:
> >> > > > > > > > >>>>>>>>>>> time_summary = {
> >> > > > > > > > >>>>>>>>>>> flag = TRUE;
> >> > > > > > > > >>>>>>>>>>> beg = "040000";
> >> > > > > > > > >>>>>>>>>>> end = "040000";
> >> > > > > > > > >>>>>>>>>>> step = 24*3600;
> >> > > > > > > > >>>>>>>>>>> width = { beg = -23*3600; end
= 0; }
> >> //
> >> > > > > > > > >>>>>>>>>>> previous
> >> > > > > > > > >>>>>>>>>>> 23-hours (05Z to 04Z)
> >> > > > > > > > >>>>>>>>>>> grib_code = [];
> >> > > > > > > > >>>>>>>>>>> obs_var = [ "COPO", "COPOPM" ];
> >> > > > > > > > >>>>>>>>>>> type = [ "max", "mean" ];
> >> > > > > > > > >>>>>>>>>>> vld_freq = 3600; // we expect
obs every
> >> hour
> >> > > > > > > > >>>>>>>>>>> vld_thresh = 0.75; // only compute
> >> summaries
> >> > > when
> >> > > > at
> >> > > > > > > > >>>>>>>>>>> least
> >> > > > > > > > >>>>>>>>>>> 75% of expect obs are present
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> That'll compute the MAX and MEAN values
at
> each
> >> > > station
> >> > > > > > > > >>>>>>>>>>> over the
> >> > > > > > > > >>>>>>>>>>> defined time period. They're
identified in
> the
> >> > > output
> >> > > > by
> >> > > > > > > > >>>>>>>>>>> the message type:
> >> > > > > > > > >>>>>>>>>>> AIRNOW_MAX_230000 or AIRNOW_MEAN_230000
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> (3) In Point-Stat config file, setup
the
> "fcst"
> >> and
> >> > > > "obs"
> >> > > > > > > > >>>>>>>>>>> sections like this for the max of the
1-hour
> >> data.
> >> > > > > > > > >>>>>>>>>>> Instead
> >> > > > > > > > >>>>>>>>>>> of specifying
> >> > > > > > > > >>>>>>>>>>> "lead_time" to filter the data, it may
be more
> >> > > > convenient
> >> > > > > > > > >>>>>>>>>>> to specify the
> >> > > > > > > > >>>>>>>>>>> valid time. So that you don't have to
modify
> >> the
> >> > > > config
> >> > > > > > > > >>>>>>>>>>> file, I'd suggest
> >> > > > > > > > >>>>>>>>>>> using environment variables:
> >> > > > > > > > >>>>>>>>>>> setenv CUR_VX_TIME 20180705_04
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> fcst = {
> >> > > > > > > > >>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>> { name = "OZMAX1"; level = "L1";
> >> > > > > > > > >>>>>>>>>>> valid_time="${CUR_VX_TIME}"; }
> >> > > > > > > > >>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>> obs = {
> >> > > > > > > > >>>>>>>>>>> convert(x) = x * 10^9;
> >> > > > > > > > >>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>> { name = "COPO"; level = "A1";
> >> message_type =
> >> > > > > > > > >>>>>>>>>>> "AIRNOW_MAX_230000"; }
> >> > > > > > > > >>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>> On Fri, Jul 6, 2018 at 12:09 PM John
Halley
> >> Gotway
> >> > <
> >> > > > > > > > >>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> Sounds good, thanks!
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>> On Fri, Jul 6, 2018 at 11:57 AM Perry
> Shafran -
> >> > NOAA
> >> > > > > > > > >>>>>>>>>>>> Affiliate <
> >> > > > > > > > >>>>>>>>>>>> perry.shafran at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> OK, then. John - for now ignore all
the bc
> >> grids
> >> > > and
> >> > > > > > > > >>>>>>>>>>>>> focus on
> >> > > > > > > > >>>>>>>>>>>>> the non-bc files. It takes some time
to
> copy
> >> > files
> >> > > > > > > > >>>>>>>>>>>>> over
> >> > > > > > > > >>>>>>>>>>>>> to Theia so we'll
> >> > > > > > > > >>>>>>>>>>>>> let him work on the files he has for
now and
> >> if
> >> > he
> >> > > > runs
> >> > > > > > > > >>>>>>>>>>>>> into problems, I'll
> >> > > > > > > > >>>>>>>>>>>>> provide him with the para5 runs (at
least
> the
> >> > 24-hr
> >> > > > > > > > >>>>>>>>>>>>> max/avg files).
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:55 PM,
> jeff.mcqueen <
> >> > > > > > > > >>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> I think the ops are ok for conus raw
> >> grids...not
> >> > > for
> >> > > > > > > > >>>>>>>>>>>>>> bias
> >> > > > > > > > >>>>>>>>>>>>>> corrected grids or for O-Conus
grids, To
> be
> >> > > safe,
> >> > > > > > > > >>>>>>>>>>>>>> you
> >> > > > > > > > >>>>>>>>>>>>>> may want to provide
> >> > > > > > > > >>>>>>>>>>>>>> him para5 runs.
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> On 7/6/2018 1:46 PM, Perry Shafran -
NOAA
> >> > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Jeff - In any case, I just sent over
to
> Theia
> >> > for
> >> > > > John
> >> > > > > > > > >>>>>>>>>>>>>> to do
> >> > > > > > > > >>>>>>>>>>>>>> some testing some production files.
Do you
> >> > think
> >> > > I
> >> > > > > > > > >>>>>>>>>>>>>> should use parallels
> >> > > > > > > > >>>>>>>>>>>>>> instead or do you think the ops
files are
> OK
> >> for
> >> > > > > > > > >>>>>>>>>>>>>> creating a MET+ use case
> >> > > > > > > > >>>>>>>>>>>>>> for AQ?
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Let me know your thoughts. I think
that
> the
> >> ops
> >> > > > files
> >> > > > > > > > >>>>>>>>>>>>>> should
> >> > > > > > > > >>>>>>>>>>>>>> be OK for now.
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 1:31 PM,
> jeff.mcqueen
> >> <
> >> > > > > > > > >>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Actually, I think I was wrong about
the
> >> > > parallels.
> >> > > > > > > > >>>>>>>>>>>>>>> We
> >> > > > > > > > >>>>>>>>>>>>>>> wanted to update the error in daily
max
> hour
> >> > > labels
> >> > > > > > > > >>>>>>>>>>>>>>> but
> >> > > > > > > > >>>>>>>>>>>>>>> management did not
> >> > > > > > > > >>>>>>>>>>>>>>> permit so we must wait until Q4FY19
CMAQ
> >> > upgrade.
> >> > > > > > > > >>>>>>>>>>>>>>> Jeff
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> On 7/6/2018 12:52 PM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Do any of the current parallels
have the
> >> > correct
> >> > > > time
> >> > > > > > > > >>>>>>>>>>>>>>> stamps? If so maybe it's better if
I send
> >> John
> >> > > > over
> >> > > > > > > > >>>>>>>>>>>>>>> parallel CMAQ data
> >> > > > > > > > >>>>>>>>>>>>>>> rather than production CMAQ.
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> I don't think that the MET team has
any
> >> other
> >> > > > > > > > >>>>>>>>>>>>>>> versions
> >> > > > > > > > >>>>>>>>>>>>>>> of
> >> > > > > > > > >>>>>>>>>>>>>>> wgrib2 than the one they already
have -
> but
> >> > they
> >> > > > can
> >> > > > > > > > >>>>>>>>>>>>>>> also clarify this as
> >> > > > > > > > >>>>>>>>>>>>>>> well. Can you clarify what the
wgrib2new
> >> does
> >> > > over
> >> > > > > > > > >>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>> current wgrib2?
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:50 PM,
> >> jeff.mcqueen <
> >> > > > > > > > >>>>>>>>>>>>>>> jeff.mcqueen at noaa.gov> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> In addition to using wgrib2new,
you may
> >> want
> >> > to
> >> > > > use
> >> > > > > > > > >>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>> latest parallel run output
(para5).
> >> > > > > > > > >>>>>>>>>>>>>>>> Regarding the 1 hour difference, I
recall
> >> > > Jianping
> >> > > > > > > > >>>>>>>>>>>>>>>> also
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that. We wanted to fix it
but
> >> > > management
> >> > > > > > > > >>>>>>>>>>>>>>>> did
> >> > > > > > > > >>>>>>>>>>>>>>>> not want to risk any
> >> > > > > > > > >>>>>>>>>>>>>>>> further delays so we will have to
do so
> >> with
> >> > the
> >> > > > > > > > >>>>>>>>>>>>>>>> FYQ4
> >> > > > > > > > >>>>>>>>>>>>>>>> implementation.
> >> > > > > > > > >>>>>>>>>>>>>>>> Jeff
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> On 7/6/2018 7:44 AM, Perry Shafran
- NOAA
> >> > > > Affiliate
> >> > > > > > > > >>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> I'm adding Jeff McQueen to the
> >> conversation.
> >> > I
> >> > > > > > > > >>>>>>>>>>>>>>>> think
> >> > > > > > > > >>>>>>>>>>>>>>>> I
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity in the
timestamps
> some
> >> > time
> >> > > > ago,
> >> > > > > > > > >>>>>>>>>>>>>>>> but I don't know if I
> >> > > > > > > > >>>>>>>>>>>>>>>> noticed that oddity with you or
with
> >> Jeff. I
> >> > > > think
> >> > > > > > > > >>>>>>>>>>>>>>>> the parallel CMAQ they
> >> > > > > > > > >>>>>>>>>>>>>>>> are running corrects that, but not
too
> >> > entirely
> >> > > > > > > > >>>>>>>>>>>>>>>> sure.
> >> > > > > > > > >>>>>>>>>>>>>>>> But he would have
> >> > > > > > > > >>>>>>>>>>>>>>>> more insight as to what is correct
and
> how
> >> to
> >> > > > > > > > >>>>>>>>>>>>>>>> approach
> >> > > > > > > > >>>>>>>>>>>>>>>> this.
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Jeff - any thoughts on the time
stamp
> >> issues
> >> > > John
> >> > > > > > > > >>>>>>>>>>>>>>>> talks
> >> > > > > > > > >>>>>>>>>>>>>>>> about?
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Thanks!
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> Perry
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>> On Thu, Jul 5, 2018 at 5:40 PM,
John
> Halley
> >> > > > Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Hi Perry,
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> I think Howard has pb2nc all
squared
> away
> >> to
> >> > > > > > > > >>>>>>>>>>>>>>>>> compute
> >> > > > > > > > >>>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>>> 24-hour maximum and average
values for
> 1-
> >> and
> >> > > 8-
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>> ozone concentrations.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> I ran pb2nc like this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> svn-met-dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >> > \
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>> PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>>>>>>>> -pbfile
> >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> And then I ran point_stat like
this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > svn-met-dev.cgd.ucar.edu/trunk/met/bin/point_stat
> >> > > > > \
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2 \
> >> > > > > > > > >>>>>>>>>>>>>>>>> pb2nc_output.nc
PointStatConfig_AIRNOW
> \
> >> > > > > > > > >>>>>>>>>>>>>>>>> -outdir point_stat -v 4
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Using the attached config files.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> That produces 1147 matched pairs:
> >> > > > > > > > >>>>>>>>>>>>>>>>> DEBUG 2: Processing OZMAX1/L1
versus
> >> COPO/A1,
> >> > > for
> >> > > > > > > > >>>>>>>>>>>>>>>>> observation type
AIRNOW_MAX_240000, over
> >> > region
> >> > > > > > > > >>>>>>>>>>>>>>>>> FULL,
> >> > > > > > > > >>>>>>>>>>>>>>>>> for interpolation
> >> > > > > > > > >>>>>>>>>>>>>>>>> method NEAREST(1), using 1147
pairs.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> But I noticed some oddities:
> >> > > > > > > > >>>>>>>>>>>>>>>>> (1) Here's how wgrib2 describes
the
> >> contents
> >> > of
> >> > > > > > > > >>>>>>>>>>>>>>>>> this
> >> > > > > > > > >>>>>>>>>>>>>>>>> "max"
> >> > > > > > > > >>>>>>>>>>>>>>>>> file:
> >> > > > > > > > >>>>>>>>>>>>>>>>> wgrib2
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > aqmmax/aqm.20180526/aqm.t12z.max_1hr_o3.148.grib2
> >> > > > > > > > >>>>>>>>>>>>>>>>> 1:0:d=2018052612:OZMAX1:1 sigma
> >> > > > > > > > >>>>>>>>>>>>>>>>> level:-2147483641--2147483618
hour ave
> >> fcst:
> >> > > > > > > > >>>>>>>>>>>>>>>>> 2:138002:d=2018052612:OZMAX1:1
sigma
> >> > > > level:17-40
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>> ave fcst:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Those timestamps look pretty
funny. But
> >> > > testing
> >> > > > > > > > >>>>>>>>>>>>>>>>> revealed
> >> > > > > > > > >>>>>>>>>>>>>>>>> that MET interprets them as being
> forecast
> >> > > hours
> >> > > > 17
> >> > > > > > > > >>>>>>>>>>>>>>>>> and 41, respectively.
> >> > > > > > > > >>>>>>>>>>>>>>>>> I think that GRIB2 file could be
encoded
> >> > > better.
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> (2) Since multiple forecast hours
are in
> >> the
> >> > > same
> >> > > > > > > > >>>>>>>>>>>>>>>>> file, we
> >> > > > > > > > >>>>>>>>>>>>>>>>> need to configure Point-Stat to
tell it
> >> which
> >> > > one
> >> > > > > > > > >>>>>>>>>>>>>>>>> to
> >> > > > > > > > >>>>>>>>>>>>>>>>> use. Do this by
> >> > > > > > > > >>>>>>>>>>>>>>>>> defining the "lead_time":
> >> > > > > > > > >>>>>>>>>>>>>>>>> fcst = {
> >> > > > > > > > >>>>>>>>>>>>>>>>> field = [
> >> > > > > > > > >>>>>>>>>>>>>>>>> { name = "OZMAX1"; level
=
> "L1";
> >> > > > lead_time
> >> > > > > > > > >>>>>>>>>>>>>>>>> =
> >> > > > > > > > >>>>>>>>>>>>>>>>> "41";' }
> >> > > > > > > > >>>>>>>>>>>>>>>>> ];
> >> > > > > > > > >>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> (3) The forecast and observation
> >> timestamps
> >> > are
> >> > > > off
> >> > > > > > > > >>>>>>>>>>>>>>>>> by 1
> >> > > > > > > > >>>>>>>>>>>>>>>>> hour.
> >> > > > > > > > >>>>>>>>>>>>>>>>> The init time of 2018052612 plus
a 41
> hour
> >> > lead
> >> > > > > > > > >>>>>>>>>>>>>>>>> time
> >> > > > > > > > >>>>>>>>>>>>>>>>> =
> >> > > > > > > > >>>>>>>>>>>>>>>>> valid time of 2018052805.
> >> > > > > > > > >>>>>>>>>>>>>>>>> We configured pb2nc to compute a
24-hour
> >> max
> >> > > > value
> >> > > > > > > > >>>>>>>>>>>>>>>>> from
> >> > > > > > > > >>>>>>>>>>>>>>>>> 04Z to 04Z. So the forecast is
valid at
> >> 05Z
> >> > > > while
> >> > > > > > > > >>>>>>>>>>>>>>>>> the obs are valid at 04Z.
> >> > > > > > > > >>>>>>>>>>>>>>>>> I account for this offset in the
> >> Point-Stat
> >> > > > config
> >> > > > > > > > >>>>>>>>>>>>>>>>> file by
> >> > > > > > > > >>>>>>>>>>>>>>>>> defining the obs_window to look
back 1
> >> hour:
> >> > > > > > > > >>>>>>>>>>>>>>>>> obs_window = {
> >> > > > > > > > >>>>>>>>>>>>>>>>> beg = -3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>> end = -3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> So what're the next steps?
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>> On Fri, Jun 29, 2018 at 5:03 PM
hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> It was caused by two input files
and
> was
> >> > > fixed.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> On 6/29/2018 11:42 AM, John
Halley
> Gotway
> >> > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> We're making progress on the
time
> >> summaries,
> >> > > > but I
> >> > > > > > > > >>>>>>>>>>>>>>>>>> found
> >> > > > > > > > >>>>>>>>>>>>>>>>>> one more issue. Please take a
look on
> >> > dakota
> >> > > > > > > > >>>>>>>>>>>>>>>>>> here:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > /d3/projects/MET/MET_Help/shafran_data_20180626
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> I ran the latest version of
pb2nc to
> >> compute
> >> > > 24-
> >> > > > > > > > >>>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>> maximum values for COPO. Then I
dumped
> >> the
> >> > > > output
> >> > > > > > > > >>>>>>>>>>>>>>>>>> to an ascii file using
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the Rscript. Then I ran these
commands
> >> to
> >> > > count
> >> > > > > > > > >>>>>>>>>>>>>>>>>> up
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the number of regular
> >> > > > > > > > >>>>>>>>>>>>>>>>>> and maximum obs values:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) egrep -v AIRNOW_MAX_240000
> >> > > pb2nc_output.txt
> >> > > > |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wc
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (2) egrep -v AIRNOW_MAX_240000
> >> > > pb2nc_output.txt
> >> > > > |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> sort -u
> >> > > > > > > > >>>>>>>>>>>>>>>>>> | wc -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (3) egrep AIRNOW_MAX_240000
> >> > pb2nc_output.txt |
> >> > > > wc
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (4) egrep AIRNOW_MAX_240000
> >> > pb2nc_output.txt |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> sort
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -u |
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wc -l
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> (1) and (2) both return
114,538...
> >> meaning
> >> > > that
> >> > > > > > > > >>>>>>>>>>>>>>>>>> all
> >> > > > > > > > >>>>>>>>>>>>>>>>>> of
> >> > > > > > > > >>>>>>>>>>>>>>>>>> the "AIRNOW" observations are
unique.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> However (3) returns 7,295 and
(4)
> returns
> >> > > 6,555
> >> > > > > > > > >>>>>>>>>>>>>>>>>> meaning
> >> > > > > > > > >>>>>>>>>>>>>>>>>> that 740 AIRNOW_MAX_240000
observations
> >> are
> >> > > > > > > > >>>>>>>>>>>>>>>>>> duplicates.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> For example, the last station ID
in
> that
> >> > file
> >> > > is
> >> > > > > > > > >>>>>>>>>>>>>>>>>> named
> >> > > > > > > > >>>>>>>>>>>>>>>>>> "2FB31EEC":
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> grep AIRNOW_MAX_240000
> pb2nc_output.txt |
> >> > grep
> >> > > > > > > > >>>>>>>>>>>>>>>>>> 2FB31EEC
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180527_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000046
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180527_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000046
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 3600 0 NA 0.000000065
> >> > > > > > > > >>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000 2FB31EEC
> >> 20180528_040000
> >> > > 32.47
> >> > > > > > > > >>>>>>>>>>>>>>>>>> -114.77
> >> > > > > > > > >>>>>>>>>>>>>>>>>> NA COPO 28800 0 NA 0.000000059
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> The obs on the first 2 lines are
> >> repeated in
> >> > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>> last two
> >> > > > > > > > >>>>>>>>>>>>>>>>>> lines.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Could you please check to see
why these
> >> > > > duplicates
> >> > > > > > > > >>>>>>>>>>>>>>>>>> are
> >> > > > > > > > >>>>>>>>>>>>>>>>>> showing up?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 5:39 PM
hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> No, I don't have them yet.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Cheers.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> On 6/28/2018 4:16 PM, John
Halley
> Gotway
> >> > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> Yep, that makes sense. There's
no
> >> reason
> >> > why
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> we'd
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> have
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> to compute a 24-hour max of AOD
values
> >> > > across 2
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> files. Do we have existing
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> examples of smaller files from
which
> we
> >> > could
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> compute a 24-hour max
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> temperature or something?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 3:33 PM
Howard
> >> Soh
> >> > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> The new unit test took about
10
> minutes
> >> > > > (550.319
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> sec).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> So pb2nc unit test took 30
minutes.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> We need smaller inputs. I will
not
> >> check
> >> > in
> >> > > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> new
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> unit test for multiple input
files.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 2:02
PM hsoh
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> <hsoh at ucar.edu>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Yes, it will change the
output (the
> >> > content
> >> > > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> obs_hid
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> variable).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> There has been an unit test
for time
> >> > > summary
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> (test
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> name: pb2nc_NDAS_summary).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> But no unit test for multiple
BURF
> >> > inputs.
> >> > > I
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> added one
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> with time summary and two
input
> files
> >> and
> >> > > > will
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> check the unit test today.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On 6/28/2018 1:35 PM, John
Halley
> >> Gotway
> >> > > > wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Ok so do you expect that this
will
> >> change
> >> > > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> nightly
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> build output? Or is this
case not
> >> > > currently
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> covered? If not, should I
add
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> it as a new unit test?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> On Thu, Jun 28, 2018 at 1:00
PM
> Howard
> >> > Soh
> >> > > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The header index was reset
for time
> >> > > summary
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> It's fixed and checked in to
SVN.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> The output is available
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> at
> >> /d3/personal/hsoh/data/pb2nc_summary/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> pb2nc_output_combined.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 27, 2018 at 2:52
PM
> John
> >> > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I saw you made some updates
to the
> >> > logic
> >> > > > for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> PB2NC.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> So I re-ran my test trying
to
> >> compute
> >> > > > 24-hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> maximum values of COPO from
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 04z to 04z. I now see
these
> obs/hdr
> >> > > > counts:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(121832
> >> > currently)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Again, I ran it through the
> Rscript
> >> to
> >> > > dump
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> ascii
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but didn't see the
> AIRNOW_MAX_240000
> >> > > string
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> show up in the output.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I can't tell if the problem
is in
> >> PB2NC
> >> > > not
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> actually
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> computing any time
summaries... or
> >> in
> >> > the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Rscript dumping the data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Are you able to take
another look
> in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > /d3/projects/MET/MET_Help/shafran_data_20180626
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
1:54 PM
> John
> >> > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Gotway <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Ok, thanks for digging
into this.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jun 26, 2018 at
1:41 PM
> >> Howard
> >> > > > Soh <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There are 57291
observations
> from
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > and
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 57246 observations
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> from
> >> > > aqm.20180528/aqm.t12z.prepbufr.tm00.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> There
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is an indexing problem
for the
> obs
> >> > > data.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I'm
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> working on this.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 25, 2018 at
2:28 PM
> >> John
> >> > > > Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Gotway
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> <johnhg at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I'm over at CG today.
Could
> you
> >> > > please
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> take
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> look at this example of
running
> >> > PB2NC.
> >> > > > I
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> copied the data over to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> /scratch/johnhg:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cd /scratch/johnhg
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
/d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > dev.cgd.ucar.edu/trunk/met/bin/pb2nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> \
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180527/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc
PB2NCConfig \
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> -pbfile
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > aqmmax/aqm.20180528/aqm.t12z.prepbufr.tm00
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> In the config file, I
request
> >> > > 24-hourly
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> maximums
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> of COPO from 04Z to 04Z,
like
> >> this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time_summary = {
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> flag = TRUE;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> beg = "040000";
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> end = "040000";
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> step = 24*3600;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> width = { beg =
> -24*3600;
> >> > end =
> >> > > > 0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> //
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> previous 24-hours
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> grib_code = [];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> obs_var = [ "COPO"
];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> type = [ "max"
];
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_freq = 0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> vld_thresh = 0.0;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> }
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> PB2NC writes 7295
> observations...
> >> > but
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 118193 headers! Which
is not
> >> what
> >> > > you'd
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> expect. The
hdr_typ_table
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> variable includes the
> >> > > > "AIRNOW_MAX_240000"
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> message type, but as you
> pointed
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> out, it is never
referenced.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I wonder if there is a
problem
> >> > caused
> >> > > by
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> using 2
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> input PrepBUFR files? I
need 2
> >> of
> >> > > them
> >> > > > to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> cover the desired 04Z to
04Z
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> time window.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Could you please take a
look?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:44 PM
> >> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Halley
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Gotway <johnhg at ucar.edu>
> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I generated it using an
AIRNOW
> >> > > PrepBUFR
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> file
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> from Perry attempting
to
> >> compute a
> >> > 24
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hour
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> maximum of obs.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Let’s debug more
together next
> >> > week.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for looking
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 22, 2018 at
6:40
> PM
> >> > hsoh
> >> > > <
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hsoh at ucar.edu> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The output from R
script is
> >> > > reasonable
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> me.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I mean there is no
> observation
> >> > data
> >> > > in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> NetCDF
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> file with the header
index of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000".
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I wonder how the
NetCDF file
> >> was
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> generated.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it the output from
pb2nc
> >> > without
> >> > > > any
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> modifications?
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's unusual that it
has MORE
> >> > > headers
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> obs
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (obs=7295 VS.
> 118193=headers).
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Dimension variables
from the
> >> > NetCDF
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> output:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nobs = UNLIMITED ; //
(7295
> >> > > currently)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr = 118193 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> nhdr_typ = 2 ;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The obs_hid contains
value
> >> from 0
> >> > > to
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3655.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The offset 1 (message
type
> for
> >> > > > summary)
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> exists
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> at the bottom of
hdr_typ
> >> variable.
> >> > > The
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> first header index for
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000" is
bigger
> >> than
> >> > > > 3655.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> So the R script seems
to be
> OK.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PS: from unit test:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript pntnc2ascii.R
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > >
> >> >
/d3/projects/MET/MET_regression/NB20180622/test_output_rev5509/pb2nc/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > ndas.20120409.t12z.prepbufr.tm00.summary.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> >
> >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cat
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > rev5509_ndas.20120409.t12z.prepbufr.tm00.summary.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | cut -f1 -d" " | sort
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> | uniq -c | less
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 6895 ADPSFC
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MAX_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_MEDIAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_MIN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338 ADPSFC_P80_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_RANGE_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 7338
ADPSFC_SDEV_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 140 ADPUPA
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MAX_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_MEDIAN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_MIN_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366 ADPUPA_P80_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_RANGE_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 8366
ADPUPA_SDEV_010000
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 15 AIRCFT
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 NULL
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 117 PROFLR
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 31 RASSDA
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 995 SFCSHP
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 249 VADWND
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 6/22/2018 4:31 PM,
John
> >> Halley
> >> > > > Gotway
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Howard,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you please take a
look on
> >> > dakota
> >> > > > in
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> here:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > /d3/projects/MET/MET_Help/shafran_data_20180622
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I ran this R-script to
dump
> it
> >> to
> >> > > > ascii:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Rscript
> >> > > > /d1/johnhg/MET/MET_development/
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> svn-met-
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > >
dev.cgd.ucar.edu/trunk/met/scripts/Rscripts/pntnc2ascii.R
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> pb2nc_output.nc >
> >> > pb2nc_output.txt
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It runs without error
but
> lists
> >> > all
> >> > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message types as
AIRNOW. But
> >> some
> >> > > of
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> should actually be
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRNOW_MAX_240000...
as
> >> evidenced
> >> > by
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this:
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ncdump -v
hdr_typ_table
> >> > > > pb2nc_output.nc
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hdr_typ_table =
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW",
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "AIRNOW_MAX_240000"
;
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> There must be some
indexing
> or
> >> > table
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> problem in the script.
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> John
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> >> > > > > > > > >>>>>>>>>>>>>>>>>>>
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: Re: PB2NC time summary questions
From: perry.shafran at noaa.gov
Time: Tue Jul 09 15:25:40 2019
I'm not sure that this has, indeed, been resolved. What's the latest?
Perry
On Tue, Jul 9, 2019 at 5:24 PM David Fillmore via RT
<met_help at ucar.edu>
wrote:
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
>
------------------------------------------------
More information about the Met_help
mailing list