[icoads_usa] Test for Tair

Shawn R. Smith smith at coaps.fsu.edu
Thu Feb 6 14:28:19 MST 2014


Hey All, 

Based on the info below, I am fine with using the internationally agreed ODS standard. Now that I see their logic that the "smaller (lower) the flag number, the more likely one will want to use the data" argument, I am fine with the flag order. I guess in some cases it is easier to code with a logic that says "use data if flag <=2" vs. "use data if flag = 1 and flag = 8" (if we applied my earlier suggestion). 

So I am voting in favor of ODS standard as we ageed in Nov.

Also agreed that we need to push the deadline to lock this down into next week. However, it would be great to have it locked down by the 14th as I will be starting to prepare my Ocean Sciences ICOADS presentation next week. FYI - recall that this will be an update on the ICOADS R3.0 progress.

Shawn

ps. not to put another bug in the works, but why did we decide to use time since 1 Jan 2014 for the IVAD date stamp instead of just sticking with the 1 Jan 1770 already in use for ICOADS? If we use the latter we need not change any imma code, though may need a slightly larger field for the IVAD date stamp (which may be the reason).

On Feb 6, 2014, at 3:57 PM, Scott Woodruff wrote:

> Eric,
> Thanks for your comments on the VQC situation. Actually I'm fine with going forward using the internationally agreed ODS primary settings, provided others agree. There appear to be strong arguments in favor, as you've emphasized. And if we go that route, that may eventually help put additional momentum behind the international standard.
> 
> Possible devil's advocate arguments though: If the standard is (arguably) flawed, do we really want to lend that support? Also since we'll have to live with what is chosen for IVAD, it seems to me this remains an important decision meriting careful final consideration (but hopefully concluded soon!).
> 
> While the ODS standards are of course reviewed (and I was involved with a few earlier proposals), I'm a bit surprised that the, to me somewhat hazy, wording for flag 9 (Used as place holder when data are missing) made it through that process.
> 
> Also, taking a look at the published standard today, I see that some justification for the ordering of flag 2 etc. is provided, which might be helpful to circulate here for purposes of this discussion:
> 
>     'Note: The quality of verified "Good" (flag 1) is considered higher (smaller flag value) compared to "Not evaluated" (flag 2), as the latter could turn out to be of any quality from good to bad, once the quality checks have been performed. Consequently, the neutral "Not evaluated" (flag 2) is placed between verified "Good" and verified "Questionable/suspect".
> [...]
>     The primary level flags are such that increasing flag values indicate decreasing data quality. This is an important property that facilitates data quality filtering and/or processing, including inheritance of quality flag values for derived variables. The quality of a calculated value inherits the lowest quality qualifier of the variables used in the calculation. For example, when we calculate density from temperature (T) and salinity (S), then if T is of “good” quality and S is of “unknown” quality, then density should inherit the “unknown quality”.'
> 
> Shawn:
> I recall Steve is away at AMS this week, so I'm wondering if it might be useful to have a small delay past your original deadline of tomorrow. Also I'm open to a telecon e.g. next week, if it would help get all these issues fully agreed. Or maybe we can reach agreement as Eric suggested via e-mail soon enough.
> 
> Attached is the document with just a small correction in my comment [8] regarding the calculation of "Julian" days (in red).       As Sandy reminded me, our lmrlib function actually calculates number of days since January 1, 1770 (versus from "January 1, 4713 BC, proleptic Julian calendar" [ref. Wikipedia]).
> 
> Sandy plans to add this new function (adopted from lmrlib) into an upcoming update of rdimma1:
>       FUNCTION GETCDN(IDAY,IMONTH,IYEAR)
> ! GET CREATION DAY NUMBER (NUMBER OF DAYS SINCE 1 JAN 2014)
> [...]
> 
> -Scott
> 
> 
> On 2/6/14 11:59 AM, Eric Freeman wrote:
>> Hey Shawn,
>> Seems we can clear this up by email, but I'm just wondering how many times we will circle around the wagon on the VQC flag scheme. We've been back and forth many times on this and need to make a decision in order to move forward.
>> 
>> While I'm not against using our own VQC 0-9 scheme, the point of using the ODS scheme 0-4,9 is so that it can more easily map with other ocean datasets.
>> I'm sure that whatever we do, the ocean community will adapt and map as best as possible, but we don't want to make it too hard to do so. 
>> ODS is an international standard, so I hate to begin to move away from that too much when consensus is growing in our groups to use such a standard.
>> 
>> 1=good, 2=questionable, 3=bad, 8=not evaluated, 9=missing 
>> While I like the order of these from Table E2, they are identical to the ODS flags, just with different numbers and placement. At that point, why rearrange the values outside of the standard? I'm sure that code can easily be written to say 2 = questionable or 3 = questionable.
>> 
>> Ultimately, we need to do what's best for ICOADS users and I'm not sure how many are actually familiar with the ODS flagging scheme, so it might not have too much of an impact (at least in the near term). As long as each flag is very clear on the meaning we should be good and I can easily accept what you and Scott have proposed.
>> 
>> Just my 2 cents.
>> Thanks,
>> Eric
>> 
>> 
>> On 2/6/2014 1:21 PM, Shawn R. Smith wrote:
>>> Hey Scott,
>>> 
>>> Attached please find Scott's latest version with my additional comments. In some cases I noted my agreement with Scott's notes within Scott's comment block (in bold text).
>>> 
>>> Is everyone comfortable with continuing to try to finalize this via email, or should we set up a brief teleconference to get this done?
>>> 
>>> Shawn
>>> 
>>> 
>>> 
>>> On Feb 5, 2014, at 6:40 PM, Scott Woodruff wrote:
>>> 
>>>> Hi Shawn and all,
>>>> 
>>>> Thanks for separating out this discussion of the IVAD attm status, from the Beaufort wind topics etc.
>>>> 
>>>> Attached is the C96/C96a document with several additional comments marked on top of your version.
>>>> 
>>>> After Sandy and I considered the detailed (including rdimma1-related software) issues today, several additional changes are suggested in Table C96 as attached (yellow highlighted, e.g. a couple of minor new field abbreviation changes). 
>>>> 
>>>> We do need a new IMMA document like the old Supplement D to start defining all the new field values, both for the IVAD attm and various other new/improved fields/attms. Thanks for offering to work on this, particularly for the IVAD portions.  -Scott
>>>> 
>>>> 
>>>> On 2/4/14 9:19 AM, Shawn R. Smith wrote:
>>>>> Hey All,
>>>>> 
>>>>> I wanted to break off the discussion of the IVAD attm status from the emails related to the Lindau correction work we are starting. Attached please find Scott's C96/C96a document with my comments. Overall, I agree that we are ready to lock down the IVAD attm format and let Sandy procede with the necessary changes to the rdimma1 code. Aside from a few minor edits that I have noted, the only big question I see is whether we have agreement on the format for the scaling factors. I have included Dave's suggestion and ask that you all give me your yea or nea on this.
>>>>> 
>>>>> Once we are in agreement, I think it would be beneficial to start a document that defines the IVAD elements and their possible values. Something akin to Supplement D: Field Configurations in the R2.5 imma documentation. This will allow us to pull all the details out of the various meeting notes and create a "how to" for IVAD developers. I am willing to work on this and can create it in the appropriate ICOADS document as needed (a start on R3.0 documentation?).
>>>>> 
>>>>> If you all can review the attached and respond with your agreement/disagreement by the end of the week (7 Feb), that would be great. 
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Shawn
>>>>> 
>>>>> Dave - if we can agree to lock this down by 7 Feb, how long will it take you to revise your code an rerun the IVAD attms for the Tair data? Looks like we are considering 1970-2007 for the prototype, but jsut having a few years would be fine. This would allow us to address Steve's comments below about a potential test to output Core+IVAD records for a user (in this case FSU). 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Jan 31, 2014, at 8:00 AM, Steve Worley wrote:
>>>>> 
>>>>>> How much different is the current Ivad format compared to the sample sent by Dave B.?
>>>>>> Is the Ivad format settled, i.e. fields name, field length, and ATTL?
>>>>>> Do we have a large enough sample for a Tair test?
>>>>>> 
>>>>>> If we have a near-final sufficient Ivad sample for Tair we could probably do something ad hoc to
>>>>>> create what Shawn needs.  By that I mean, ingest the Ivad into the IVAD-DB and write
>>>>>> out all the Tair Ivad  and the necessary parts of the Core in a one-off ASCII records so 
>>>>>> the science test can go forward.  Certainly, if the write IMMA1 software is ready to handle
>>>>>> Ivad then we could remove the ad hoc nature of this test.
>>>>>> 
>>>>>> Thanks
>>>>>> Steve
>>>>>> 
>>>>>> 
>>>>>> On Jan 30, 2014, at 2:58 PM, Shawn R. Smith <smith at coaps.fsu.edu> wrote:
>>>>>> 
>>>>>>> Hey Steve,
>>>>>>> 
>>>>>>> As Scott noted, yes it was Bill Murray at CPO I was thinking about. As for opening a discussion, I would like to start by asking him if he sees any future opportunities in his program for IVAD. I never had a chance to ask him why they rejected our LOI, but I expect it has to do with program priorities (maybe they are getting too land focused). Either way, it would be good to know for the future.
>>>>>>> 
>>>>>>> For part 3, the scientific IVAD demonstration, we do have some ideas along that line and plan to complete something before our present IVAD funding terminates (in August or maybe later if we get a no-cost extension). Ideas at present include:
>>>>>>> 
>>>>>>> 1. Calculating psuedo-fluxes (like those in da silva) with and without the corrections in the IVAD attm. 
>>>>>>> 2. Comparing adjusted and unadjusted winds to satellite wind values
>>>>>>> 
>>>>>>> What would really help us exercise the science would be to have a mechanism whereby we can extract both the ICOADS core record and the IVAD attm for a set of UIDs. As it is now, we have sample IVAD attm from Dave, but do not have the associated core records. We would have to develop a code to search through the IMMA2.5.1records to find the matching core records for each UID. If there is a way we could use the IVAD at NCAR to make this easier, that would be great (and would be a much better demonstration of the power of the tool). I assume this would require Dave to create a year (or so) of IVAD attm that we could import into the IVAD. Then we would just need a mechanism to export "all" records with an IVAD attm along with the associated core record (or for that matter, just export the full IMMA1 record for the subset that have IVAD attms).
>>>>>>> 
>>>>>>> I certainly would like to discus this further. I will be focusing more time in IVAD in the next month or so (in prep for Ocean Sciences and CLIMAR4), so now would be a great time.
>>>>>>> 
>>>>>>> Let us keep the discussion going. Feel free to call me if you want to discuss this further on Friday.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> Shawn
>>>>>>> 
>>>>>>> On Jan 28, 2014, at 4:27 PM, Steve Worley wrote:
>>>>>>> 
>>>>>>>> Shawn,
>>>>>>>> 
>>>>>>>> What thoughts do you have in opening a discussion with CPO?  Our schedule
>>>>>>>> fell into disarray, and CPO rejected our recent LOI.  I sure would like to see the
>>>>>>>> 3rd part, scientific IVAD demonstration, be done, but I have no ideas.  Once the 
>>>>>>>> IMMA formatting has been finalized, the DB should work will be trivial.  The GUI
>>>>>>>> to IVAD will take some work, careful documentation, and testing.  
>>>>>>>> 
>>>>>>>> Steve
>>>>>>>> 
>>>>>>>> On Jan 28, 2014, at 12:16 PM, Scott Woodruff <scott.d.woodruff at noaa.gov> wrote:
>>>>>>>> 
>>>>>>>>> Eric,
>>>>>>>>> Thanks for drafting these. Attached includes a number of suggestions/questions you can consider, building on top of Shawn's version.
>>>>>>>>> 
>>>>>>>>> While hopefully we will have the bulk of the IVAD infrastructure (e.g. at NCAR in the DBMS; and in terms of the supporting IMMA format and software revisions) in place around the time the funding runs out, I agree with Shawn that we need to discuss with CPO sometime (note: possibly Bill Murray was intended?) future funding prospects. -Scott
>>>>>>>> 
>>>>>>>> ---
>>>>>>>> Steven Worley / NCAR
>>>>>>>> worley at ucar.edu
>>>>>>>> rda.ucar.edu
>>>>>>>> Wrk: 303.497.1248
>>>>>>>> Mobile: 720.468.1961
>>>>>>>> ----
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> icoads_usa mailing list
>>>>>>>> icoads_usa at mailman.ucar.edu
>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>>>>> 
>>>>>> 
>>>>>> ---
>>>>>> Steven Worley / NCAR
>>>>>> worley at ucar.edu
>>>>>> rda.ucar.edu
>>>>>> Wrk: 303.497.1248
>>>>>> Mobile: 720.468.1961
>>>>>> ----
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> icoads_usa mailing list
>>>>> icoads_usa at mailman.ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>> 
>>>> 
>>>> <IMMA-Rev-v15_WORKING-TablesC96-C96a_srs-sdw.docx>
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> icoads_usa mailing list
>>> icoads_usa at mailman.ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>> 
> 
> <IMMA-Rev-v15_WORKING-TablesC96-C96a_srs-sdw-srs-sdw.docx>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140206/78143c2b/attachment-0001.html 


More information about the icoads_usa mailing list