[icoads_usa] Test for Tair

Scott Woodruff scott.d.woodruff at noaa.gov
Thu Feb 6 13:57:38 MST 2014


Eric,
Thanks for your comments on the VQC situation. Actually I'm fine with 
going forward using the internationally agreed ODS primary settings, 
provided others agree. There appear to be strong arguments in favor, as 
you've emphasized. And if we go that route, that may eventually help put 
additional momentum behind the international standard.

Possible devil's advocate arguments though: If the standard is 
(arguably) flawed, do we really want to lend that support? Also since 
we'll have to live with what is chosen for IVAD, it seems to me this 
remains an important decision meriting careful final consideration (but 
hopefully concluded soon!).

While the ODS standards are of course reviewed (and I was involved with 
a few earlier proposals), I'm a bit surprised that the, to me somewhat 
hazy, wording for flag 9 (/Used as place holder when data are missing/) 
made it through that process.

Also, taking a look at the published standard today, I see that some 
justification for the ordering of flag 2 etc. is provided, which might 
be helpful to circulate here for purposes of this discussion:

/    'Note: The quality of verified "Good" (flag 1) is considered higher 
(smaller flag value) compared to "Not evaluated" (flag 2), as the latter 
could turn out to be of any quality from good to bad, once the quality 
checks have been performed. Consequently, the neutral "Not evaluated" 
(flag 2) is placed between verified "Good" and verified 
"Questionable/suspect".
[...]
     The primary level flags are such that increasing flag values 
indicate decreasing data quality. This is an important property that 
facilitates data quality filtering and/or processing, including 
inheritance of quality flag values for derived variables. The quality of 
a calculated value inherits the lowest quality qualifier of the 
variables used in the calculation. For example, when we calculate 
density from temperature (T) and salinity (S), then if T is of "good" 
quality and S is of "unknown" quality, then density should inherit the 
"unknown quality".'
/
Shawn:
I recall Steve is away at AMS this week, so I'm wondering if it might be 
useful to have a small delay past your original deadline of tomorrow. 
Also I'm open to a telecon e.g. next week, if it would help get all 
these issues fully agreed. Or maybe we can reach agreement as Eric 
suggested via e-mail soon enough.

Attached is the document with just a small correction in my comment [8] 
regarding the calculation of "Julian" days (in red). As Sandy reminded 
me, our lmrlib function actually calculates number of days since January 
1, 1770 (versus from "January 1, 4713 BC, proleptic Julian calendar" 
[ref. Wikipedia]).

Sandy plans to add this new function (adopted from lmrlib) into an 
upcoming update of rdimma1:
       FUNCTION GETCDN(IDAY,IMONTH,IYEAR)
! GET CREATION DAY NUMBER (NUMBER OF DAYS SINCE 1 JAN 2014)
[...]

-Scott


On 2/6/14 11:59 AM, Eric Freeman wrote:
> Hey Shawn,
> Seems we can clear this up by email, but I'm just wondering how many 
> times we will circle around the wagon on the VQC flag scheme. We've 
> been back and forth many times on this and need to make a decision in 
> order to move forward.
>
> While I'm not against using our own VQC 0-9 scheme, the point of using 
> the ODS scheme 0-4,9 is so that it can more easily map with other 
> ocean datasets.
> I'm sure that whatever we do, the ocean community will adapt and map 
> as best as possible, but we don't want to make it too hard to do so.
> ODS is an international standard, so I hate to begin to move away from 
> that too much when consensus is growing in our groups to use such a 
> standard.
>
> 1=good, 2=questionable, 3=bad, 8=not evaluated, 9=missing
> While I like the order of these from Table E2, they are identical to 
> the ODS flags, just with different numbers and placement. At that 
> point, why rearrange the values outside of the standard? I'm sure that 
> code can easily be written to say 2 = questionable or 3 = questionable.
>
> Ultimately, we need to do what's best for ICOADS users and I'm not 
> sure how many are actually familiar with the ODS flagging scheme, so 
> it might not have too much of an impact (at least in the near term). 
> As long as each flag is very clear on the meaning we should be good 
> and I can easily accept what you and Scott have proposed.
>
> Just my 2 cents.
> Thanks,
> Eric
>
>
> On 2/6/2014 1:21 PM, Shawn R. Smith wrote:
>> Hey Scott,
>>
>> Attached please find Scott's latest version with my additional 
>> comments. In some cases I noted my agreement with Scott's notes 
>> within Scott's comment block (in bold text).
>>
>> Is everyone comfortable with continuing to try to finalize this via 
>> email, or should we set up a brief teleconference to get this done?
>>
>> Shawn
>>
>>
>>
>> On Feb 5, 2014, at 6:40 PM, Scott Woodruff wrote:
>>
>>> Hi Shawn and all,
>>>
>>> Thanks for separating out this discussion of the IVAD attm status, 
>>> from the Beaufort wind topics etc.
>>>
>>> Attached is the C96/C96a document with several additional comments 
>>> marked on top of your version.
>>>
>>> After Sandy and I considered the detailed (including rdimma1-related 
>>> software) issues today, several additional changes are suggested in 
>>> Table C96 as attached (yellow highlighted, e.g. a couple of minor 
>>> new field abbreviation changes).
>>>
>>> We do need a new IMMA document like the old Supplement D to start 
>>> defining all the new field values, both for the IVAD attm and 
>>> various other new/improved fields/attms. Thanks for offering to work 
>>> on this, particularly for the IVAD portions.  -Scott
>>>
>>>
>>> On 2/4/14 9:19 AM, Shawn R. Smith wrote:
>>>> Hey All,
>>>>
>>>> I wanted to break off the discussion of the IVAD attm status from 
>>>> the emails related to the Lindau correction work we are starting. 
>>>> Attached please find Scott's C96/C96a document with my comments. 
>>>> Overall, I agree that we are ready to lock down the IVAD attm 
>>>> format and let Sandy procede with the necessary changes to the 
>>>> rdimma1 code. Aside from a few minor edits that I have noted, the 
>>>> only big question I see is whether we have agreement on the format 
>>>> for the scaling factors. I have included Dave's suggestion and ask 
>>>> that you all give me your yea or nea on this.
>>>>
>>>> Once we are in agreement, I think it would be beneficial to start a 
>>>> document that defines the IVAD elements and their possible values. 
>>>> Something akin to Supplement D: Field Configurations in the R2.5 
>>>> imma documentation. This will allow us to pull all the details out 
>>>> of the various meeting notes and create a "how to" for IVAD 
>>>> developers. I am willing to work on this and can create it in the 
>>>> appropriate ICOADS document as needed (a start on R3.0 documentation?).
>>>>
>>>> If you all can review the attached and respond with your 
>>>> agreement/disagreement by the end of the week (7 Feb), that would 
>>>> be great.
>>>>
>>>> Thanks,
>>>>
>>>> Shawn
>>>>
>>>> Dave - if we can agree to lock this down by 7 Feb, how long will it 
>>>> take you to revise your code an rerun the IVAD attms for the Tair 
>>>> data? Looks like we are considering 1970-2007 for the prototype, 
>>>> but jsut having a few years would be fine. This would allow us to 
>>>> address Steve's comments below about a potential test to output 
>>>> Core+IVAD records for a user (in this case FSU).
>>>>
>>>>
>>>>
>>>> On Jan 31, 2014, at 8:00 AM, Steve Worley wrote:
>>>>
>>>>> How much different is the current /Ivad /format compared to the 
>>>>> sample sent by Dave B.?
>>>>> Is the/Ivad /format settled, i.e. fields name, field length, and ATTL?
>>>>> Do we have a large enough sample for a Tair test?
>>>>>
>>>>> If we have a near-final sufficient /Ivad /sample for Tair we could 
>>>>> probably do something /ad hoc/ to
>>>>> create what Shawn needs.  By that I mean, ingest the /Ivad /into 
>>>>> the IVAD-DB and write
>>>>> out all the Tair /Ivad/  and the necessary parts of the /Core/ in 
>>>>> a one-off ASCII records so
>>>>> the science test can go forward.  Certainly, if the write IMMA1 
>>>>> software is ready to handle
>>>>> /Ivad /then we could remove the /ad hoc /nature of this test.
>>>>>
>>>>> Thanks
>>>>> Steve
>>>>>
>>>>>
>>>>> On Jan 30, 2014, at 2:58 PM, Shawn R. Smith <smith at coaps.fsu.edu 
>>>>> <mailto:smith at coaps.fsu.edu>> wrote:
>>>>>
>>>>>> Hey Steve,
>>>>>>
>>>>>> As Scott noted, yes it was Bill Murray at CPO I was thinking 
>>>>>> about. As for opening a discussion, I would like to start by 
>>>>>> asking him if he sees any future opportunities in his program for 
>>>>>> IVAD. I never had a chance to ask him why they rejected our LOI, 
>>>>>> but I expect it has to do with program priorities (maybe they are 
>>>>>> getting too land focused). Either way, it would be good to know 
>>>>>> for the future.
>>>>>>
>>>>>> For part 3, the scientific IVAD demonstration, we do have some 
>>>>>> ideas along that line and plan to complete something before our 
>>>>>> present IVAD funding terminates (in August or maybe later if we 
>>>>>> get a no-cost extension). Ideas at present include:
>>>>>>
>>>>>> 1. Calculating psuedo-fluxes (like those in da silva) with and 
>>>>>> without the corrections in the IVAD attm.
>>>>>> 2. Comparing adjusted and unadjusted winds to satellite wind values
>>>>>>
>>>>>> What would really help us exercise the science would be to have a 
>>>>>> mechanism whereby we can extract both the ICOADS core record and 
>>>>>> the IVAD attm for a set of UIDs. As it is now, we have sample 
>>>>>> IVAD attm from Dave, but do not have the associated core records. 
>>>>>> We would have to develop a code to search through the 
>>>>>> IMMA2.5.1records to find the matching core records for each UID. 
>>>>>> If there is a way we could use the IVAD at NCAR to make this 
>>>>>> easier, that would be great (and would be a much better 
>>>>>> demonstration of the power of the tool). I assume this would 
>>>>>> require Dave to create a year (or so) of IVAD attm that we could 
>>>>>> import into the IVAD. Then we would just need a mechanism to 
>>>>>> export "all" records with an IVAD attm along with the associated 
>>>>>> core record (or for that matter, just export the full IMMA1 
>>>>>> record for the subset that have IVAD attms).
>>>>>>
>>>>>> I certainly would like to discus this further. I will be focusing 
>>>>>> more time in IVAD in the next month or so (in prep for Ocean 
>>>>>> Sciences and CLIMAR4), so now would be a great time.
>>>>>>
>>>>>> Let us keep the discussion going. Feel free to call me if you 
>>>>>> want to discuss this further on Friday.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Shawn
>>>>>>
>>>>>> On Jan 28, 2014, at 4:27 PM, Steve Worley wrote:
>>>>>>
>>>>>>> Shawn,
>>>>>>>
>>>>>>> What thoughts do you have in opening a discussion with CPO?  Our 
>>>>>>> schedule
>>>>>>> fell into disarray, and CPO rejected our recent LOI.  I sure 
>>>>>>> would like to see the
>>>>>>> 3rd part, scientific IVAD demonstration, be done, but I have no 
>>>>>>> ideas.  Once the
>>>>>>> IMMA formatting has been finalized, the DB should work will be 
>>>>>>> trivial.  The GUI
>>>>>>> to IVAD will take some work, careful documentation, and testing.
>>>>>>>
>>>>>>> Steve
>>>>>>>
>>>>>>> On Jan 28, 2014, at 12:16 PM, Scott Woodruff 
>>>>>>> <scott.d.woodruff at noaa.gov <mailto:scott.d.woodruff at noaa.gov>> 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Eric,
>>>>>>>> Thanks for drafting these. Attached includes a number of 
>>>>>>>> suggestions/questions you can consider, building on top of 
>>>>>>>> Shawn's version.
>>>>>>>>
>>>>>>>> While hopefully we will have the bulk of the IVAD 
>>>>>>>> infrastructure (e.g. at NCAR in the DBMS; and in terms of the 
>>>>>>>> supporting IMMA format and software revisions) in place around 
>>>>>>>> the time the funding runs out, I agree with Shawn that we need 
>>>>>>>> to discuss with CPO sometime (note: possibly_Bill_Murray was 
>>>>>>>> intended?) future funding prospects. -Scott
>>>>>>>
>>>>>>> ---
>>>>>>> Steven Worley / NCAR
>>>>>>> worley at ucar.edu <mailto:worley at ucar.edu>
>>>>>>> rda.ucar.edu <http://rda.ucar.edu/>
>>>>>>> Wrk: 303.497.1248
>>>>>>> Mobile: 720.468.1961
>>>>>>> ----
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> icoads_usa mailing list
>>>>>>> icoads_usa at mailman.ucar.edu <mailto:icoads_usa at mailman.ucar.edu>
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>>>>
>>>>>
>>>>> ---
>>>>> Steven Worley / NCAR
>>>>> worley at ucar.edu <mailto:worley at ucar.edu>
>>>>> rda.ucar.edu <http://rda.ucar.edu/>
>>>>> Wrk: 303.497.1248
>>>>> Mobile: 720.468.1961
>>>>> ----
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> icoads_usa mailing list
>>>> icoads_usa at mailman.ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>>>
>>>
>>> <IMMA-Rev-v15_WORKING-TablesC96-C96a_srs-sdw.docx>
>>
>>
>>
>> _______________________________________________
>> icoads_usa mailing list
>> icoads_usa at mailman.ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/icoads_usa
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140206/f541ccf4/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMMA-Rev-v15_WORKING-TablesC96-C96a_srs-sdw-srs-sdw.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 57178 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140206/f541ccf4/attachment-0001.bin 


More information about the icoads_usa mailing list