[icoads_usa] Test for Tair

Eric Freeman eric.freeman at noaa.gov
Thu Feb 6 14:25:37 MST 2014

Hi Scott,
More very good points on this. Careful consideration is needed as this 
will be part of IVAD for some time, possibly.
I'm happy to set up a telecon if needed next week when Steve is back in 
the office.
On 2/6/2014 3:57 PM, Scott Woodruff wrote:
> Eric,
> Thanks for your comments on the VQC situation. Actually I'm fine with 
> going forward using the internationally agreed ODS primary settings, 
> provided others agree. There appear to be strong arguments in favor, 
> as you've emphasized. And if we go that route, that may eventually 
> help put additional momentum behind the international standard.
> Possible devil's advocate arguments though: If the standard is 
> (arguably) flawed, do we really want to lend that support? Also since 
> we'll have to live with what is chosen for IVAD, it seems to me this 
> remains an important decision meriting careful final consideration 
> (but hopefully concluded soon!).
> While the ODS standards are of course reviewed (and I was involved 
> with a few earlier proposals), I'm a bit surprised that the, to me 
> somewhat hazy, wording for flag 9 (/Used as place holder when data are 
> missing/) made it through that process.
> Also, taking a look at the published standard today, I see that some 
> justification for the ordering of flag 2 etc. is provided, which might 
> be helpful to circulate here for purposes of this discussion:
> /    'Note: The quality of verified "Good" (flag 1) is considered 
> higher (smaller flag value) compared to "Not evaluated" (flag 2), as 
> the latter could turn out to be of any quality from good to bad, once 
> the quality checks have been performed. Consequently, the neutral "Not 
> evaluated" (flag 2) is placed between verified "Good" and verified 
> "Questionable/suspect".
> [...]
>     The primary level flags are such that increasing flag values 
> indicate decreasing data quality. This is an important property that 
> facilitates data quality filtering and/or processing, including 
> inheritance of quality flag values for derived variables. The quality 
> of a calculated value inherits the lowest quality qualifier of the 
> variables used in the calculation. For example, when we calculate 
> density from temperature (T) and salinity (S), then if T is of "good" 
> quality and S is of "unknown" quality, then density should inherit the 
> "unknown quality".'
> /
> Shawn:
> I recall Steve is away at AMS this week, so I'm wondering if it might 
> be useful to have a small delay past your original deadline of 
> tomorrow. Also I'm open to a telecon e.g. next week, if it would help 
> get all these issues fully agreed. Or maybe we can reach agreement as 
> Eric suggested via e-mail soon enough.
> Attached is the document with just a small correction in my comment 
> [8] regarding the calculation of "Julian" days (in red). As Sandy 
> reminded me, our lmrlib function actually calculates number of days 
> since January 1, 1770 (versus from "January 1, 4713 BC, proleptic 
> Julian calendar" [ref. Wikipedia]).
> Sandy plans to add this new function (adopted from lmrlib) into an 
> upcoming update of rdimma1:
> [...]
> -Scott
> On 2/6/14 11:59 AM, Eric Freeman wrote:
>> Hey Shawn,
>> Seems we can clear this up by email, but I'm just wondering how many 
>> times we will circle around the wagon on the VQC flag scheme. We've 
>> been back and forth many times on this and need to make a decision in 
>> order to move forward.
>> While I'm not against using our own VQC 0-9 scheme, the point of 
>> using the ODS scheme 0-4,9 is so that it can more easily map with 
>> other ocean datasets.
>> I'm sure that whatever we do, the ocean community will adapt and map 
>> as best as possible, but we don't want to make it too hard to do so.
>> ODS is an international standard, so I hate to begin to move away 
>> from that too much when consensus is growing in our groups to use 
>> such a standard.
>> 1=good, 2=questionable, 3=bad, 8=not evaluated, 9=missing
>> While I like the order of these from Table E2, they are identical to 
>> the ODS flags, just with different numbers and placement. At that 
>> point, why rearrange the values outside of the standard? I'm sure 
>> that code can easily be written to say 2 = questionable or 3 = 
>> questionable.
>> Ultimately, we need to do what's best for ICOADS users and I'm not 
>> sure how many are actually familiar with the ODS flagging scheme, so 
>> it might not have too much of an impact (at least in the near term). 
>> As long as each flag is very clear on the meaning we should be good 
>> and I can easily accept what you and Scott have proposed.
>> Just my 2 cents.
>> Thanks,
>> Eric
>> On 2/6/2014 1:21 PM, Shawn R. Smith wrote:
>>> Hey Scott,
>>> Attached please find Scott's latest version with my additional 
>>> comments. In some cases I noted my agreement with Scott's notes 
>>> within Scott's comment block (in bold text).
>>> Is everyone comfortable with continuing to try to finalize this via 
>>> email, or should we set up a brief teleconference to get this done?
>>> Shawn
>>> On Feb 5, 2014, at 6:40 PM, Scott Woodruff wrote:
>>>> Hi Shawn and all,
>>>> Thanks for separating out this discussion of the IVAD attm status, 
>>>> from the Beaufort wind topics etc.
>>>> Attached is the C96/C96a document with several additional comments 
>>>> marked on top of your version.
>>>> After Sandy and I considered the detailed (including 
>>>> rdimma1-related software) issues today, several additional changes 
>>>> are suggested in Table C96 as attached (yellow highlighted, e.g. a 
>>>> couple of minor new field abbreviation changes).
>>>> We do need a new IMMA document like the old Supplement D to start 
>>>> defining all the new field values, both for the IVAD attm and 
>>>> various other new/improved fields/attms. Thanks for offering to 
>>>> work on this, particularly for the IVAD portions. -Scott
>>>> On 2/4/14 9:19 AM, Shawn R. Smith wrote:
>>>>> Hey All,
>>>>> I wanted to break off the discussion of the IVAD attm status from 
>>>>> the emails related to the Lindau correction work we are starting. 
>>>>> Attached please find Scott's C96/C96a document with my comments. 
>>>>> Overall, I agree that we are ready to lock down the IVAD attm 
>>>>> format and let Sandy procede with the necessary changes to the 
>>>>> rdimma1 code. Aside from a few minor edits that I have noted, the 
>>>>> only big question I see is whether we have agreement on the format 
>>>>> for the scaling factors. I have included Dave's suggestion and ask 
>>>>> that you all give me your yea or nea on this.
>>>>> Once we are in agreement, I think it would be beneficial to start 
>>>>> a document that defines the IVAD elements and their possible 
>>>>> values. Something akin to Supplement D: Field Configurations in 
>>>>> the R2.5 imma documentation. This will allow us to pull all the 
>>>>> details out of the various meeting notes and create a "how to" for 
>>>>> IVAD developers. I am willing to work on this and can create it in 
>>>>> the appropriate ICOADS document as needed (a start on R3.0 
>>>>> documentation?).
>>>>> If you all can review the attached and respond with your 
>>>>> agreement/disagreement by the end of the week (7 Feb), that would 
>>>>> be great.
>>>>> Thanks,
>>>>> Shawn
>>>>> Dave - if we can agree to lock this down by 7 Feb, how long will 
>>>>> it take you to revise your code an rerun the IVAD attms for the 
>>>>> Tair data? Looks like we are considering 1970-2007 for the 
>>>>> prototype, but jsut having a few years would be fine. This would 
>>>>> allow us to address Steve's comments below about a potential test 
>>>>> to output Core+IVAD records for a user (in this case FSU).
>>>>> On Jan 31, 2014, at 8:00 AM, Steve Worley wrote:
>>>>>> How much different is the current /Ivad /format compared to the 
>>>>>> sample sent by Dave B.?
>>>>>> Is the/Ivad /format settled, i.e. fields name, field length, and 
>>>>>> ATTL?
>>>>>> Do we have a large enough sample for a Tair test?
>>>>>> If we have a near-final sufficient /Ivad /sample for Tair we 
>>>>>> could probably do something /ad hoc/ to
>>>>>> create what Shawn needs.  By that I mean, ingest the /Ivad /into 
>>>>>> the IVAD-DB and write
>>>>>> out all the Tair /Ivad/  and the necessary parts of the /Core/ in 
>>>>>> a one-off ASCII records so
>>>>>> the science test can go forward.  Certainly, if the write IMMA1 
>>>>>> software is ready to handle
>>>>>> /Ivad /then we could remove the /ad hoc /nature of this test.
>>>>>> Thanks
>>>>>> Steve
>>>>>> On Jan 30, 2014, at 2:58 PM, Shawn R. Smith <smith at coaps.fsu.edu 
>>>>>> <mailto:smith at coaps.fsu.edu>> wrote:
>>>>>>> Hey Steve,
>>>>>>> As Scott noted, yes it was Bill Murray at CPO I was thinking 
>>>>>>> about. As for opening a discussion, I would like to start by 
>>>>>>> asking him if he sees any future opportunities in his program 
>>>>>>> for IVAD. I never had a chance to ask him why they rejected our 
>>>>>>> LOI, but I expect it has to do with program priorities (maybe 
>>>>>>> they are getting too land focused). Either way, it would be good 
>>>>>>> to know for the future.
>>>>>>> For part 3, the scientific IVAD demonstration, we do have some 
>>>>>>> ideas along that line and plan to complete something before our 
>>>>>>> present IVAD funding terminates (in August or maybe later if we 
>>>>>>> get a no-cost extension). Ideas at present include:
>>>>>>> 1. Calculating psuedo-fluxes (like those in da silva) with and 
>>>>>>> without the corrections in the IVAD attm.
>>>>>>> 2. Comparing adjusted and unadjusted winds to satellite wind values
>>>>>>> What would really help us exercise the science would be to have 
>>>>>>> a mechanism whereby we can extract both the ICOADS core record 
>>>>>>> and the IVAD attm for a set of UIDs. As it is now, we have 
>>>>>>> sample IVAD attm from Dave, but do not have the associated core 
>>>>>>> records. We would have to develop a code to search through the 
>>>>>>> IMMA2.5.1records to find the matching core records for each UID. 
>>>>>>> If there is a way we could use the IVAD at NCAR to make this 
>>>>>>> easier, that would be great (and would be a much better 
>>>>>>> demonstration of the power of the tool). I assume this would 
>>>>>>> require Dave to create a year (or so) of IVAD attm that we could 
>>>>>>> import into the IVAD. Then we would just need a mechanism to 
>>>>>>> export "all" records with an IVAD attm along with the associated 
>>>>>>> core record (or for that matter, just export the full IMMA1 
>>>>>>> record for the subset that have IVAD attms).
>>>>>>> I certainly would like to discus this further. I will be 
>>>>>>> focusing more time in IVAD in the next month or so (in prep for 
>>>>>>> Ocean Sciences and CLIMAR4), so now would be a great time.
>>>>>>> Let us keep the discussion going. Feel free to call me if you 
>>>>>>> want to discuss this further on Friday.
>>>>>>> Cheers,
>>>>>>> Shawn
>>>>>>> On Jan 28, 2014, at 4:27 PM, Steve Worley wrote:
>>>>>>>> Shawn,
>>>>>>>> What thoughts do you have in opening a discussion with CPO? 
>>>>>>>>  Our schedule
>>>>>>>> fell into disarray, and CPO rejected our recent LOI.  I sure 
>>>>>>>> would like to see the
>>>>>>>> 3rd part, scientific IVAD demonstration, be done, but I have no 
>>>>>>>> ideas.  Once the
>>>>>>>> IMMA formatting has been finalized, the DB should work will be 
>>>>>>>> trivial.  The GUI
>>>>>>>> to IVAD will take some work, careful documentation, and testing.
>>>>>>>> Steve
>>>>>>>> On Jan 28, 2014, at 12:16 PM, Scott Woodruff 
>>>>>>>> <scott.d.woodruff at noaa.gov <mailto:scott.d.woodruff at noaa.gov>> 
>>>>>>>> wrote:
>>>>>>>>> Eric,
>>>>>>>>> Thanks for drafting these. Attached includes a number of 
>>>>>>>>> suggestions/questions you can consider, building on top of 
>>>>>>>>> Shawn's version.
>>>>>>>>> While hopefully we will have the bulk of the IVAD 
>>>>>>>>> infrastructure (e.g. at NCAR in the DBMS; and in terms of the 
>>>>>>>>> supporting IMMA format and software revisions) in place around 
>>>>>>>>> the time the funding runs out, I agree with Shawn that we need 
>>>>>>>>> to discuss with CPO sometime (note: possibly_Bill_Murray was 
>>>>>>>>> intended?) future funding prospects. -Scott
>>>>>>>> ---
>>>>>>>> Steven Worley / NCAR
>>>>>>>> worley at ucar.edu <mailto:worley at ucar.edu>
>>>>>>>> rda.ucar.edu <http://rda.ucar.edu/>
>>>>>>>> Wrk: 303.497.1248
>>>>>>>> Mobile: 720.468.1961
>>>>>>>> ----
>>>>>>>> _______________________________________________
>>>>>>>> icoads_usa mailing list
>>>>>>>> icoads_usa at mailman.ucar.edu <mailto:icoads_usa at mailman.ucar.edu>
>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>>>> ---
>>>>>> Steven Worley / NCAR
>>>>>> worley at ucar.edu <mailto:worley at ucar.edu>
>>>>>> rda.ucar.edu <http://rda.ucar.edu/>
>>>>>> Wrk: 303.497.1248
>>>>>> Mobile: 720.468.1961
>>>>>> ----
>>>>> _______________________________________________
>>>>> icoads_usa mailing list
>>>>> icoads_usa at mailman.ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>> <IMMA-Rev-v15_WORKING-TablesC96-C96a_srs-sdw.docx>
>>> _______________________________________________
>>> icoads_usa mailing list
>>> icoads_usa at mailman.ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
> _______________________________________________
> icoads_usa mailing list
> icoads_usa at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/icoads_usa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140206/e75b9a72/attachment-0001.html 

More information about the icoads_usa mailing list