[icoads_usa] IMMA1 data issue!

Steve Worley worley at ucar.edu
Thu May 8 10:34:06 MDT 2014


Just to clarify:

The erroneous days (e.g. 31 September) are NOT adjusted on the IMMA records, Hua 
adjusts an internal DB index to insure the record will always be output in the appropriate
month, i.e. in September for the case above.  His index is build from year-month-day, therefore
a valid day must be used.  DB's have very robust, and useful, date functions.

In the same way, if day is missing, the IMMA records in not altered, a realistic day value 
(1 is what Hua uses) is used to build the DB index.  

These adjustments are critical for the DB to function accurately when producing monthly IMMA1
output files, and eventually for user queries that will have resolution to the day.

Steve




On May 8, 2014, at 8:15 AM, Zaihua Ji <zji at ucar.edu> wrote:

> Steve,
> 
> the refilling IVADDB is done for the full records unto 2007. Here the to log files:
> 4228 dates_changed.txt
> 69268 missing_dy.txt
> 
> under http://rda.ucar.edu/zji/IVADDB/
> 
> I am filling the IVAD and ECR atoms now.
> 
> Hua
> 
> On May 4, 2014, at 5:56 PM, Zaihua Ji <zji at ucar.edu> wrote:
> 
>> Steve,
>> 
>> I am reloading IVADDB with fixed dates if field dy is missing or out of ranges. It is 70% done at the moment.
>> 
>> I have some reports dumped under http://rda.ucar.edu/zji/IVADDB/
>> 
>> There seem a lot of dy issues even for the modern dates.
>> 
>> I'll refresh the reports when the loading is finished.
>> 
>> At the moment, I want generate output monthly files identical to the original ones unless with additional IVAD records. We can deal with dy issues further later.
>> 
>> Hua
>> 
>> 
>> 
>> On May 4, 2014, at 4:37 PM, Steve Worley <worley at ucar.edu> wrote:
>> 
>>> Scott,
>>> 
>>> Let me speak with Hua before we take any action.  We need an effective path of least resistance.
>>> Hua is preparing the IVAD-DB workflow to produce IMMA1 multi-record output.  Shawn needs
>>> R2.5.1 with Ivad attachments, soon,  for some analysis prior to CLIMAR-IV.
>>> 
>>> Is it safe to assume that the first public release of IVAD will be based on Rel_3.0?
>>> 
>>> If yes:
>>>  - Shawn's test data will be for the modern period and will not be significantly affected by
>>>    these few early period erroneous day dates.
>>>  - Maybe Hua can simply reject these records for the time being, since we are still in a testing
>>>    phase and only team-based research is taking place. 
>>> 
>>> We can then prepare a tighter screening of day date for Rel_3.0 and fix the problem there.  All
>>> Ivad attachments received, including the ECR main record addition, will need to be re-integrated
>>> in the IVAD-DB.  Probably they should also be re-computed by the international contributors.
>>>  
>>> 
>>> Thanks
>>> Steve
>>> 
>>> 
>>> 
>>> On May 4, 2014, at 3:57 PM, Scott Woodruff <scott.d.woodruff at noaa.gov> wrote:
>>> 
>>>> Steve,
>>>> A quick response (I'll be back in the office on Tue.) but a few comments/questions we can discuss further sometime:
>>>> 
>>>> (a) First, I wondered about the practicality of determining the overall extent of the illegal DY=31 problem throughout R2.5.1?
>>>> 
>>>> We know the problem also exists, to a small extent (est. 22 reports) in the US Maury Collection (as Boyin at NCDC discovered one such case recently), at least partly arising (if I understand correctly from Sandy) during the complicated post-translation time adjustment step:
>>>>      http://icoads.noaa.gov/maury_time.html
>>>> 
>>>> (b) Assuming the problem is very limited in extent (small number of reports, early time periods only?) what would be the pros/cons of simply deleting all those reports, and thus generating a R2.5.2?
>>>> 
>>>> Our current approach of not worrying much about the illegal dates goes back to early COADS processing ideas, with monthly summaries then perceived to be more important. Clearly now that users are often more interested in the IMMA individual obs. scale, these illegal dates (while flagged by NCDC-QC) reflect unfavorably on our processing. 
>>>> 
>>>> (c) Then for R3.0 we could put on the list for consideration of feasibility of possibly fixing these limited cases and re-introducing the records.
>>>> 
>>>> For the US Maury, there is at least one important additional issue for consideration: the early SLP biases--possibly also applicable to the new (to be included in R3.0) German Maury Collection, ref. Fig. 3 at:
>>>>      http://icoads.noaa.gov/maury_german.html
>>>> (note: US Maury was originally translated into LMR, not IMMA, and therefore there may be other format weaknesses/omissions that could be addressed, in the event of a reprocessing/re-translation).
>>>> 
>>>> -Scott
>>>> 
>>>> 
>>>> On 5/1/14, 9:54 AM, Steve Worley wrote:
>>>>> Scott,
>>>>> 
>>>>> This date issue, although minimal in extent, could create some strange anomalies.  If the IMMA1 records created
>>>>> from the ivaddb dump are not the same as created with the normal release processing more questions will come up.
>>>>> Do we need to permit invalid days to be output from the ivaddb?  Currently, I believe the user selection process in the
>>>>> GUI only allows a month granularity selection - it would be appropriate to offer more, say to the day level (as Hua notes).  
>>>>> 
>>>>> My feeling now, and I welcome your thoughts, is that we should sort out the position, date, and time at the record level
>>>>> as best we can.  I know historically it does not affect the MSG products.  
>>>>> 
>>>>> Steve
>>>>> 
>>>>> On May 1, 2014, at 10:39 AM, Sandra Lubker <sandra.j.lubker at noaa.gov> wrote:
>>>>> 
>>>>>> Hua,
>>>>>> 
>>>>>> I like 2. fake the date value to Sep. 30.
>>>>>> 
>>>>>> Sandy
>>>>>> 
>>>>>> On 5/1/14 10:31 AM, Zaihua Ji wrote:
>>>>>>> Sandy,
>>>>>>> 
>>>>>>> I have field type date for each record in ivaddb. The Sep. 31 is an invalid value. Here are several ways to handle it. Please suggest which way should go:
>>>>>>> 1. keep the current logic. Sep. 31 will be Oct. 1
>>>>>>> 2. fake the date value to Sep. 30.
>>>>>>> 3. Tread it as missing dy. (Sep. 1 will be used for this case for date value, I use dy as 1 if dy is missing)
>>>>>>> 
>>>>>>> I need the date values for query on subletting in the future too.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Hua
>>>>>>> 
>>>>>>> On May 1, 2014, at 10:19 AM, Sandra Lubker <Sandra.J.Lubker at noaa.gov> wrote:
>>>>>>> 
>>>>>>>> Hi Hua,
>>>>>>>> 
>>>>>>>> Any way to always allow 31 days per month?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Sandy
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 5/1/14 8:58 AM, Zaihua Ji wrote:
>>>>>>>>> Hi Sandy,
>>>>>>>>> 
>>>>>>>>> I am testing my code to dump IMMA1 data from ivaddb to monthly files. I noticed a problem for the last record in R2.5.1.1899.09:
>>>>>>>>> 1899 9312300-3200  1400 1302                 11356 26      10110      139          10 1502                   165 46724156  4   0       H8          111111F71AAA11AA11AAA     981505NO8O2512199 0 025  7
>>>>>>>>> 
>>>>>>>>> The day of 31 for September is out of range. My code automatically set the date of this record to 1899-10-01. When I write the data back to monthly files, this record is written into file of 1899-10 instead.
>>>>>>>>> 
>>>>>>>>> What do you expect me to do with this type of records?
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> hua
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> ---
>>>>> Steven Worley / NCAR
>>>>> worley at ucar.edu
>>>>> rda.ucar.edu
>>>>> Wrk: 303.497.1248
>>>>> Mobile: 720.468.1961
>>>>> ----
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>> ---
>>> Steven Worley / NCAR
>>> worley at ucar.edu
>>> rda.ucar.edu
>>> Wrk: 303.497.1248
>>> Mobile: 720.468.1961
>>> ----
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> icoads_usa mailing list
>>> icoads_usa at mailman.ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>> 
> 

---
Steven Worley / NCAR
worley at ucar.edu
rda.ucar.edu
Wrk: 303.497.1248
Mobile: 720.468.1961
----




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140508/9d02b175/attachment.html 


More information about the icoads_usa mailing list