[icoads_usa] ICOADS 3.0 plans

Atkinson, Chris chris.atkinson at metoffice.gov.uk
Thu Nov 28 09:30:20 MST 2013

Steve, Eric, Scott,

Sorry for such a slow reply, I've been completely tied up with a few deadlines the last few weeks and had to put off replying to various emails.

Nice speaking to everyone a few weeks back in the teleconference.  I found the update on ICOADS 3.0 plans very helpful.

Some follow up points on the topics we've been discussing below:

- UIDs
Great to hear the UIDs will be preserved for ICOADS 3.0.0.  As a bit of further feedback, I see that in the UID section of the telecon notes it states that we 'Need to be able to identify records that have been changed from one release to the next'.  I'm not sure I said anything at this point during the telecon but I would agree with this.  For example, if we were to link QC outcomes or bias corrections via UID to an ob, it may be these value added data are only relevant to a particular version of that ob - it would therefore be important that a new UID is allocated if a record changes.

As mentioned in the telecon, unfortunately we don't have the resources at present to help with the problem of gathering and converting moored buoy data to IMMA.  What I should have added at the time is that we are planning to look into gathering GTMBA data as part of some work we're doing for ESA, but I don't think this will happen until 2015 and so it's not helpful for the ICOADS 3.0.0 plans.  However, if you think this could be in anyway helpful in future then perhaps we could discuss?

- Drifting buoys
My guess is that AOML SVP should comprise about 80% of the ISDM archive, as SVP drifters form the bulk of the global drifter array.  But this would need confirming.  I'm not sure about data formats having not downloaded data from the ISDM archive before.

My suspicion is that the AOML SVP data will offer a higher quality starting point than the ISDM version - it certainly sounds that way from the documentation I've collected - but I think it would be prudent to discuss this with AOML and ISDM before any decision is taken for ICOADS.  Possibly Rick Lumpkin and Mayra Pazos from AOML, and Mathieu Ouellet from ISDM might be helpful people to contact?  It would certainly be nice to get a conversation going about how the drifter data archives and ICOADS could benefit by pooling our knowledge.

The QC outcomes from my work are already mapped to UIDs and so we could certainly discuss feeding them in via IVAD.  But I think engaging with AOML and ISDM is the way forward for ICOADS because (i) we do not have dedicated resources for updating our QC'd drifter archive (whereas AOML and ISDM are dedicated drifter data centres) (ii) our automated QC approach inevitably suffers from some good data failing QC and some bad data passing QC, whilst AOML and ISDM scrutinise each new months data manually (I believe) and so in theory should suffer less from this issue.

As one final, small bit of feedback, I also noticed whilst looking over the ISDM website that ISDM add a couple of extra numbers to drifter WMO IDs to help distinguish between drifters whose WMO ID has been re-used.  By the sounds of it AOML also do a similar thing.  I don't think ICOADS preserve this info in the ID field but it might be helpful to have this in future for tracking individual platforms.  I believe this is less of a problem in recent years because of the switch to 7-character WMO IDs.

Thanks, Chris.

Chris Atkinson  Oceans Observations Research Scientist
Met Office Hadley Centre  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
Tel: +44 (0)1392 886120  Fax: +44 (0)1392 885681
Email: chris.atkinson at metoffice.gov.uk  Website: www.metoffice.gov.uk

From: Steve Worley [mailto:worley at ucar.edu]
Sent: 01 November 2013 16:34
To: Atkinson, Chris
Cc: icoads_usa at mailman.ucar.edu
Subject: Re: ICOADS 3.0 plans


Just to add to Eric's response.

It is our full intent to preserve the UIDs you currently have in Rel 2.5.1 as we
go forward to Release 3.0.0.  The UIDs are assigned at one level prior to the duplicate
elimination and the data preconditioning step. In the same place you acquired Rel 2.5.1 you will see
Rel 2.5.1i (the "i" stands for intermediate, http://rda.ucar.edu/datasets/ds540.0/#!access) - these
are the foundation data reports.  If we tweak the data processing rules, which is one
way we improve the quality of the final data, a few records from Rel 2.5.1 may not appear in Rel 3.0.0,
but they will in general be replaced with another record having a different UID.

Currently, we have not identified resources to process the GTMBA to the fullest extend possible - like we
did for Release 2.5.  This means going to NOAA and JAMSTEC websites, finding the bulk data to download,
understanding the various formats, converting the data and metadata into IMMA1 format.  A few particulars
for the record:
 - For Rel. 2.5 we possessed 4 GTMBA sources ( hourly and 10-minute TOA data from NOAA, real-time hourly
   averaged data from JAMSTEC, hourly SLP data from NOAA).  The 10-minute sample data were sub-sampled to
   hourly when Release 2.5 was produced.
   {We could consider changing this practice to better support the satellite community.  Note to ICOADS PIs - could not
     we move the sub-sampling to a pre-conditioning step prior to creating the monthly summary statistics and carry the
     full resolution in the observational archive?}
 - RAMA data was not available when we did the delayed mode data processing for Rel 2.5, it should be included the
   next time we process the GTMBA
 - The World Ocean Database contains only the daily means from TAO, PIRATA, and RAMA.  Not JAMSTEC, not
    high resolution (hourly or 10-minute), and I'm not certain about the special SLP data file we found and included in R2.5

On drifting buoy data:
Eric gave a good summary and these are some interesting ideas.
- First I do like the idea of gaining a higher quality starting point.  What portion of the ISDM collection is
  included in the AOML SVP archive?  Could we have a stroke of luck and benefit from both of these sources being
  in the same data format?  This would minimize the effort required to include AOML SVP.
- Are your QC and bias discoveries mappable to UIDs, i.e. individual records in ICOADS?  This would make
  enable the IVAD approach for improving the data quality.

A long reach question:  If the we were able to achieve the change noted in red (above),  would you or someone you
know in the satellite community be willing to convert these research moored buoys to IMMA data format for Release 3.0?

Thanks Chris.  We appreciate the way you use and work on these data.  It helps us understand the user community and
forms the way we go forward.


On Oct 28, 2013, at 12:35 PM, Eric Freeman <Eric.Freeman at noaa.gov<mailto:Eric.Freeman at noaa.gov>> wrote:

Hi Chris,
Good to hear from you! Things are going pretty good here, especially for Scott. He's on his annual holiday and expected back in the office next week.
Hope you are doing well!

Your questions below cross many different topics the US partners are trying to deal with and NCAR has been the key player in many of these. In the future, please include Steve Worley (cc'd), and between the 3 of us we should be able to answer most of your questions.

My answers are interleaved below. I may have to defer to Steve on some of these.


- For the database I'm creating I've made use of the UIDs released recently in IMMA1 format, which I've been referring to as ICOADS 2.5.1 (hopefully correctly!).  I was wondering whether the observations in ICOADS 3.0 will preserve the ICOADS 2.5.1 UIDs?  I appreciate the concept of the UID is one UID per ob consistent across ICOADS updates, but I wasn't sure if those released in ICOADS 2.5.1 were preliminary.

R2.5.1 is a preliminary product to help us sort this UID and IVAD process out, as well as upgrades to IMMA(1) for R3.0.
I'm not certain if these UID's will carry over or if we will start fresh. Steve - can you elaborate on this?

- Are there any plans for ICOADS 3.0 to make use of high resolution GTMBA data?  This is something that we are being pushed on for satellite SST validation purposes but which is not straightforward to address as, as I'm sure you appreciate, the GTMBA archive is quite fragmented.  It's my understanding that at present ICOADS gathers the high-resolution buoy data but sub-samples to hourly resolution.  I was also wondering what the planned data coverage for ICOADS 3.0 is in terms of TAO/TRITON, PIRATA and RAMA arrays?  If I recall correctly, I think the RAMA data in ICOADS 2.5 is limited.

The GTMBA data are very fragmented and hard to collect given their scattered structure. Hi frequency data is becoming more common, whether through RVs or buoy observations (e.g. Arctic Shell moored buoys in the Arctic), so it is something we definitely have to deal with. An example is the SAMOS RV data from FSU. They subsample to hourly.
Steve - are the buoys in WOD13 being added to the Nocn attm subsampled to hourly in cases of high frequency? Plans to grab additional GTMBA data not part of WOD13?

- I was thinking again about drifting buoys.  Looking at the ICOADS 3.0 preliminary workplan it looks like an update of the ISDM deck is planned, but I've been wondering how I can make the most of my recent investigations into drifting buoy SST data.  I guess a conclusion of this work was that various gross errors are still apparent in delayed mode ICOADS 2.5 drifter data (biases, enhanced noise, buoys out of water etc.) and although I made an attempt to develop QC procedures that address this, this is likely a one-off piece of work and not something we can keep revisiting for future ICOADS releases.  As such, I was wondering how such errors might be minimised in the future.  Two suggestions jump to mind: (i) feeding back our findings to ISDM via ICOADS to see if improvements could be made to the ISDM QC procedures and archive (ii) incorporating the AOML SVP archive held at ISDM into ICOADS; I believe many of these errors are removed by AOML QC (and SVP buoys are approx. 80% of the global array).  This is only something I've been pondering lately but I'd be interested to know what you think.

It would be great if you could provide any bias adjustments or corrections for the IVAD project where your work has pointed out some of the mentioned obvious issues. We hope you will consider that as we move forward with that product.
We rely heavily on user feedback and would be willing to work with ISDM and you to provide feedback on their data quality and potential qc enhancements they can put to use in the future.
The AOML SVP product should also be considered as a new data source for ICOADS, considering the higher quality data and qc procedures  already applied to the data.

I know my answers are a bit vague, but hopefully Steve can fill in some gaps, and Scott will be back just before our meeting, which looks to be on Wed 6 Nov, and can likely elaborate.

Eric Freeman
Marine Observation Analyst
Team STG, Inc./ERT - Government Contractor

Ingest and Analysis Branch
Global Climate Applications Division
NOAA's National Climatic Data Center
151 Patton Avenue, Asheville, NC 28801-5001
Phone: +1 828.271.4463
Fax:   +1 828.271.4022
Eric.Freeman at noaa.gov<mailto:Eric.Freeman at noaa.gov>

Steven Worley / NCAR
worley at ucar.edu<mailto:worley at ucar.edu>
Wrk: 303.497.1248
Mobile: 720.468.1961

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20131128/33deaf0f/attachment-0001.html 

More information about the icoads_usa mailing list