[icoads_usa] Test for Tair

Scott Woodruff scott.d.woodruff at noaa.gov
Thu Feb 6 15:04:45 MST 2014


Hi Shawn and all,

Looks like most of the votes are already in (w/ Dave's positive vote as 
well) and so most likely we will stick with the original Nov. decision 
to use the ODS primary flag scheme (& sorry to be probably the only bug 
in the works on that question!). Thus I expect we will certainly be able 
to lock down this and related questions before the 14th.

Regarding your ps:
> ps. not to put another bug in the works, but why did we decide to use 
> time since 1 Jan 2014 for the IVAD date stamp instead of just sticking 
> with the 1 Jan 1770 already in use for ICOADS? If we use the latter we 
> need not change any imma code, though may need a slightly larger field 
> for the IVAD date stamp (which may be the reason).
I checked the minutes, and the change to 1 Jan 2014 was also agreed at 
our Nov. Boulder meeting (vs. previous zero day of 1 Oct 2012 for old 
field ADJN).

I think that the arbitrary 1 Jan 1770 is just associated with the lmrlib 
routine {ixdtnd}, it's not otherwise a fixed internal feature of ICOADS. 
The lmrlib software I believe we set up that way before any data prior 
to 1770 (now back to 1662) were included in ICOADS, and now we might 
wish the starting date in that software was set earlier.  Also I think 
having more day space available in the CDNI 3-char field is essential 
actually, referring back to the footnote:

ZZZ is46655 in base36, thus the use of three characters allows almost 
128 years of entries (in contrast, ZZ is 1295 in base36, thus two 
characters would allow only ~3.5 years of entries).


Thanks for the FYI, so I expect we'll also be discussing with you next 
week regarding more general R3.0 progress/issues.  -Scott


On 2/6/14 2:28 PM, Shawn R. Smith wrote:
> Hey All,
>
> Based on the info below, I am fine with using the internationally 
> agreed ODS standard. Now that I see their logic that the "smaller 
> (lower) the flag number, the more likely one will want to use the 
> data" argument, I am fine with the flag order. I guess in some cases 
> it is easier to code with a logic that says "use data if flag <=2" vs. 
> "use data if flag = 1 and flag = 8" (if we applied my earlier 
> suggestion).
>
> So I am voting in favor of ODS standard as we ageed in Nov.
>
> Also agreed that we need to push the deadline to lock this down into 
> next week. However, it would be great to have it locked down by the 
> 14th as I will be starting to prepare my Ocean Sciences ICOADS 
> presentation next week. FYI - recall that this will be an update on 
> the ICOADS R3.0 progress.
>
> Shawn
>
> ps. not to put another bug in the works, but why did we decide to use 
> time since 1 Jan 2014 for the IVAD date stamp instead of just sticking 
> with the 1 Jan 1770 already in use for ICOADS? If we use the latter we 
> need not change any imma code, though may need a slightly larger field 
> for the IVAD date stamp (which may be the reason).
>
> On Feb 6, 2014, at 3:57 PM, Scott Woodruff wrote:
>
>> Eric,
>> Thanks for your comments on the VQC situation. Actually I'm fine with 
>> going forward using the internationally agreed ODS primary settings, 
>> provided others agree. There appear to be strong arguments in favor, 
>> as you've emphasized. And if we go that route, that may eventually 
>> help put additional momentum behind the international standard.
>>
>> Possible devil's advocate arguments though: If the standard is 
>> (arguably) flawed, do we really want to lend that support? Also since 
>> we'll have to live with what is chosen for IVAD, it seems to me this 
>> remains an important decision meriting careful final consideration 
>> (but hopefully concluded soon!).
>>
>> While the ODS standards are of course reviewed (and I was involved 
>> with a few earlier proposals), I'm a bit surprised that the, to me 
>> somewhat hazy, wording for flag 9 (/Used as place holder when data 
>> are missing/) made it through that process.
>>
>> Also, taking a look at the published standard today, I see that some 
>> justification for the ordering of flag 2 etc. is provided, which 
>> might be helpful to circulate here for purposes of this discussion:
>>
>> /    'Note: The quality of verified "Good" (flag 1) is considered 
>> higher (smaller flag value) compared to "Not evaluated" (flag 2), as 
>> the latter could turn out to be of any quality from good to bad, once 
>> the quality checks have been performed. Consequently, the neutral 
>> "Not evaluated" (flag 2) is placed between verified "Good" and 
>> verified "Questionable/suspect".
>> [...]
>>     The primary level flags are such that increasing flag values 
>> indicate decreasing data quality. This is an important property that 
>> facilitates data quality filtering and/or processing, including 
>> inheritance of quality flag values for derived variables. The quality 
>> of a calculated value inherits the lowest quality qualifier of the 
>> variables used in the calculation. For example, when we calculate 
>> density from temperature (T) and salinity (S), then if T is of “good” 
>> quality and S is of “unknown” quality, then density should inherit 
>> the “unknown quality”.'
>> /
>> Shawn:
>> I recall Steve is away at AMS this week, so I'm wondering if it might 
>> be useful to have a small delay past your original deadline of 
>> tomorrow. Also I'm open to a telecon e.g. next week, if it would help 
>> get all these issues fully agreed. Or maybe we can reach agreement as 
>> Eric suggested via e-mail soon enough.
>>
>> Attached is the document with just a small correction in my comment 
>> [8] regarding the calculation of "Julian" days (in red). As Sandy 
>> reminded me, our lmrlib function actually calculates number of days 
>> since January 1, 1770 (versus from "January 1, 4713 BC, proleptic 
>> Julian calendar" [ref. Wikipedia]).
>>
>> Sandy plans to add this new function (adopted from lmrlib) into an 
>> upcoming update of rdimma1:
>>       FUNCTION GETCDN(IDAY,IMONTH,IYEAR)
>> ! GET CREATION DAY NUMBER (NUMBER OF DAYS SINCE 1 JAN 2014)
>> [...]
>>
>> -Scott
>>
>>
>> On 2/6/14 11:59 AM, Eric Freeman wrote:
>>> Hey Shawn,
>>> Seems we can clear this up by email, but I'm just wondering how many 
>>> times we will circle around the wagon on the VQC flag scheme. We've 
>>> been back and forth many times on this and need to make a decision 
>>> in order to move forward.
>>>
>>> While I'm not against using our own VQC 0-9 scheme, the point of 
>>> using the ODS scheme 0-4,9 is so that it can more easily map with 
>>> other ocean datasets.
>>> I'm sure that whatever we do, the ocean community will adapt and map 
>>> as best as possible, but we don't want to make it too hard to do so.
>>> ODS is an international standard, so I hate to begin to move away 
>>> from that too much when consensus is growing in our groups to use 
>>> such a standard.
>>>
>>> 1=good, 2=questionable, 3=bad, 8=not evaluated, 9=missing
>>> While I like the order of these from Table E2, they are identical to 
>>> the ODS flags, just with different numbers and placement. At that 
>>> point, why rearrange the values outside of the standard? I'm sure 
>>> that code can easily be written to say 2 = questionable or 3 = 
>>> questionable.
>>>
>>> Ultimately, we need to do what's best for ICOADS users and I'm not 
>>> sure how many are actually familiar with the ODS flagging scheme, so 
>>> it might not have too much of an impact (at least in the near term). 
>>> As long as each flag is very clear on the meaning we should be good 
>>> and I can easily accept what you and Scott have proposed.
>>>
>>> Just my 2 cents.
>>> Thanks,
>>> Eric
>>>
>>>
>>> On 2/6/2014 1:21 PM, Shawn R. Smith wrote:
>>>> Hey Scott,
>>>>
>>>> Attached please find Scott's latest version with my additional 
>>>> comments. In some cases I noted my agreement with Scott's notes 
>>>> within Scott's comment block (in bold text).
>>>>
>>>> Is everyone comfortable with continuing to try to finalize this via 
>>>> email, or should we set up a brief teleconference to get this done?
>>>>
>>>> Shawn
>>>>
>>>>
>>>>
>>>> On Feb 5, 2014, at 6:40 PM, Scott Woodruff wrote:
>>>>
>>>>> Hi Shawn and all,
>>>>>
>>>>> Thanks for separating out this discussion of the IVAD attm status, 
>>>>> from the Beaufort wind topics etc.
>>>>>
>>>>> Attached is the C96/C96a document with several additional comments 
>>>>> marked on top of your version.
>>>>>
>>>>> After Sandy and I considered the detailed (including 
>>>>> rdimma1-related software) issues today, several additional changes 
>>>>> are suggested in Table C96 as attached (yellow highlighted, e.g. a 
>>>>> couple of minor new field abbreviation changes).
>>>>>
>>>>> We do need a new IMMA document like the old Supplement D to start 
>>>>> defining all the new field values, both for the IVAD attm and 
>>>>> various other new/improved fields/attms. Thanks for offering to 
>>>>> work on this, particularly for the IVAD portions.  -Scott
>>>>>
>>>>>
>>>>> On 2/4/14 9:19 AM, Shawn R. Smith wrote:
>>>>>> Hey All,
>>>>>>
>>>>>> I wanted to break off the discussion of the IVAD attm status from 
>>>>>> the emails related to the Lindau correction work we are starting. 
>>>>>> Attached please find Scott's C96/C96a document with my comments. 
>>>>>> Overall, I agree that we are ready to lock down the IVAD attm 
>>>>>> format and let Sandy procede with the necessary changes to the 
>>>>>> rdimma1 code. Aside from a few minor edits that I have noted, the 
>>>>>> only big question I see is whether we have agreement on the 
>>>>>> format for the scaling factors. I have included Dave's suggestion 
>>>>>> and ask that you all give me your yea or nea on this.
>>>>>>
>>>>>> Once we are in agreement, I think it would be beneficial to start 
>>>>>> a document that defines the IVAD elements and their possible 
>>>>>> values. Something akin to Supplement D: Field Configurations in 
>>>>>> the R2.5 imma documentation. This will allow us to pull all the 
>>>>>> details out of the various meeting notes and create a "how to" 
>>>>>> for IVAD developers. I am willing to work on this and can create 
>>>>>> it in the appropriate ICOADS document as needed (a start on R3.0 
>>>>>> documentation?).
>>>>>>
>>>>>> If you all can review the attached and respond with your 
>>>>>> agreement/disagreement by the end of the week (7 Feb), that would 
>>>>>> be great.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Shawn
>>>>>>
>>>>>> Dave - if we can agree to lock this down by 7 Feb, how long will 
>>>>>> it take you to revise your code an rerun the IVAD attms for the 
>>>>>> Tair data? Looks like we are considering 1970-2007 for the 
>>>>>> prototype, but jsut having a few years would be fine. This would 
>>>>>> allow us to address Steve's comments below about a potential test 
>>>>>> to output Core+IVAD records for a user (in this case FSU).
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Jan 31, 2014, at 8:00 AM, Steve Worley wrote:
>>>>>>
>>>>>>> How much different is the current /Ivad /format compared to the 
>>>>>>> sample sent by Dave B.?
>>>>>>> Is the/Ivad /format settled, i.e. fields name, field length, and 
>>>>>>> ATTL?
>>>>>>> Do we have a large enough sample for a Tair test?
>>>>>>>
>>>>>>> If we have a near-final sufficient /Ivad /sample for Tair we 
>>>>>>> could probably do something /ad hoc/ to
>>>>>>> create what Shawn needs.  By that I mean, ingest the /Ivad /into 
>>>>>>> the IVAD-DB and write
>>>>>>> out all the Tair /Ivad/ and the necessary parts of the /Core/ in 
>>>>>>> a one-off ASCII records so
>>>>>>> the science test can go forward.  Certainly, if the write IMMA1 
>>>>>>> software is ready to handle
>>>>>>> /Ivad /then we could remove the /ad hoc /nature of this test.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Steve
>>>>>>>
>>>>>>>
>>>>>>> On Jan 30, 2014, at 2:58 PM, Shawn R. Smith <smith at coaps.fsu.edu 
>>>>>>> <mailto:smith at coaps.fsu.edu>> wrote:
>>>>>>>
>>>>>>>> Hey Steve,
>>>>>>>>
>>>>>>>> As Scott noted, yes it was Bill Murray at CPO I was thinking 
>>>>>>>> about. As for opening a discussion, I would like to start by 
>>>>>>>> asking him if he sees any future opportunities in his program 
>>>>>>>> for IVAD. I never had a chance to ask him why they rejected our 
>>>>>>>> LOI, but I expect it has to do with program priorities (maybe 
>>>>>>>> they are getting too land focused). Either way, it would be 
>>>>>>>> good to know for the future.
>>>>>>>>
>>>>>>>> For part 3, the scientific IVAD demonstration, we do have some 
>>>>>>>> ideas along that line and plan to complete something before our 
>>>>>>>> present IVAD funding terminates (in August or maybe later if we 
>>>>>>>> get a no-cost extension). Ideas at present include:
>>>>>>>>
>>>>>>>> 1. Calculating psuedo-fluxes (like those in da silva) with and 
>>>>>>>> without the corrections in the IVAD attm.
>>>>>>>> 2. Comparing adjusted and unadjusted winds to satellite wind values
>>>>>>>>
>>>>>>>> What would really help us exercise the science would be to have 
>>>>>>>> a mechanism whereby we can extract both the ICOADS core record 
>>>>>>>> and the IVAD attm for a set of UIDs. As it is now, we have 
>>>>>>>> sample IVAD attm from Dave, but do not have the associated core 
>>>>>>>> records. We would have to develop a code to search through the 
>>>>>>>> IMMA2.5.1records to find the matching core records for each 
>>>>>>>> UID. If there is a way we could use the IVAD at NCAR to make 
>>>>>>>> this easier, that would be great (and would be a much better 
>>>>>>>> demonstration of the power of the tool). I assume this would 
>>>>>>>> require Dave to create a year (or so) of IVAD attm that we 
>>>>>>>> could import into the IVAD. Then we would just need a mechanism 
>>>>>>>> to export "all" records with an IVAD attm along with the 
>>>>>>>> associated core record (or for that matter, just export the 
>>>>>>>> full IMMA1 record for the subset that have IVAD attms).
>>>>>>>>
>>>>>>>> I certainly would like to discus this further. I will be 
>>>>>>>> focusing more time in IVAD in the next month or so (in prep for 
>>>>>>>> Ocean Sciences and CLIMAR4), so now would be a great time.
>>>>>>>>
>>>>>>>> Let us keep the discussion going. Feel free to call me if you 
>>>>>>>> want to discuss this further on Friday.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Shawn
>>>>>>>>
>>>>>>>> On Jan 28, 2014, at 4:27 PM, Steve Worley wrote:
>>>>>>>>
>>>>>>>>> Shawn,
>>>>>>>>>
>>>>>>>>> What thoughts do you have in opening a discussion with CPO? 
>>>>>>>>>  Our schedule
>>>>>>>>> fell into disarray, and CPO rejected our recent LOI.  I sure 
>>>>>>>>> would like to see the
>>>>>>>>> 3rd part, scientific IVAD demonstration, be done, but I have 
>>>>>>>>> no ideas.  Once the
>>>>>>>>> IMMA formatting has been finalized, the DB should work will be 
>>>>>>>>> trivial.  The GUI
>>>>>>>>> to IVAD will take some work, careful documentation, and testing.
>>>>>>>>>
>>>>>>>>> Steve
>>>>>>>>>
>>>>>>>>> On Jan 28, 2014, at 12:16 PM, Scott Woodruff 
>>>>>>>>> <scott.d.woodruff at noaa.gov <mailto:scott.d.woodruff at noaa.gov>> 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Eric,
>>>>>>>>>> Thanks for drafting these. Attached includes a number of 
>>>>>>>>>> suggestions/questions you can consider, building on top of 
>>>>>>>>>> Shawn's version.
>>>>>>>>>>
>>>>>>>>>> While hopefully we will have the bulk of the IVAD 
>>>>>>>>>> infrastructure (e.g. at NCAR in the DBMS; and in terms of the 
>>>>>>>>>> supporting IMMA format and software revisions) in place 
>>>>>>>>>> around the time the funding runs out, I agree with Shawn that 
>>>>>>>>>> we need to discuss with CPO sometime (note: 
>>>>>>>>>> possibly_Bill_Murray was intended?) future funding prospects. 
>>>>>>>>>> -Scott
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Steven Worley / NCAR
>>>>>>>>> worley at ucar.edu <mailto:worley at ucar.edu>
>>>>>>>>> rda.ucar.edu <http://rda.ucar.edu/>
>>>>>>>>> Wrk: 303.497.1248
>>>>>>>>> Mobile: 720.468.1961
>>>>>>>>> ----
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> icoads_usa mailing list
>>>>>>>>> icoads_usa at mailman.ucar.edu <mailto:icoads_usa at mailman.ucar.edu>
>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Steven Worley / NCAR
>>>>>>> worley at ucar.edu <mailto:worley at ucar.edu>
>>>>>>> rda.ucar.edu <http://rda.ucar.edu/>
>>>>>>> Wrk: 303.497.1248
>>>>>>> Mobile: 720.468.1961
>>>>>>> ----
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> icoads_usa mailing list
>>>>>> icoads_usa at mailman.ucar.edu
>>>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>>>
>>>>>
>>>>> <IMMA-Rev-v15_WORKING-TablesC96-C96a_srs-sdw.docx>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> icoads_usa mailing list
>>>> icoads_usa at mailman.ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>
>>>
>>
>> <IMMA-Rev-v15_WORKING-TablesC96-C96a_srs-sdw-srs-sdw.docx>
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140206/b0586129/attachment-0001.html 


More information about the icoads_usa mailing list