[icoads_usa] ICOADS 3.0 plans

Steve Worley worley at ucar.edu
Fri Nov 1 10:33:51 MDT 2013


Just to add to Eric's response.

It is our full intent to preserve the UIDs you currently have in Rel 2.5.1 as we
go forward to Release 3.0.0.  The UIDs are assigned at one level prior to the duplicate 
elimination and the data preconditioning step. In the same place you acquired Rel 2.5.1 you will see 
Rel 2.5.1i (the "i" stands for intermediate, http://rda.ucar.edu/datasets/ds540.0/#!access) - these
are the foundation data reports.  If we tweak the data processing rules, which is one 
way we improve the quality of the final data, a few records from Rel 2.5.1 may not appear in Rel 3.0.0, 
but they will in general be replaced with another record having a different UID.  

Currently, we have not identified resources to process the GTMBA to the fullest extend possible - like we
did for Release 2.5.  This means going to NOAA and JAMSTEC websites, finding the bulk data to download,
understanding the various formats, converting the data and metadata into IMMA1 format.  A few particulars
for the record:
 - For Rel. 2.5 we possessed 4 GTMBA sources ( hourly and 10-minute TOA data from NOAA, real-time hourly
   averaged data from JAMSTEC, hourly SLP data from NOAA).  The 10-minute sample data were sub-sampled to
   hourly when Release 2.5 was produced.
   {We could consider changing this practice to better support the satellite community.  Note to ICOADS PIs - could not 
     we move the sub-sampling to a pre-conditioning step prior to creating the monthly summary statistics and carry the
     full resolution in the observational archive?}
 - RAMA data was not available when we did the delayed mode data processing for Rel 2.5, it should be included the
   next time we process the GTMBA
 - The World Ocean Database contains only the daily means from TAO, PIRATA, and RAMA.  Not JAMSTEC, not
    high resolution (hourly or 10-minute), and I'm not certain about the special SLP data file we found and included in R2.5

On drifting buoy data:
Eric gave a good summary and these are some interesting ideas.
- First I do like the idea of gaining a higher quality starting point.  What portion of the ISDM collection is 
  included in the AOML SVP archive?  Could we have a stroke of luck and benefit from both of these sources being
  in the same data format?  This would minimize the effort required to include AOML SVP.
- Are your QC and bias discoveries mappable to UIDs, i.e. individual records in ICOADS?  This would make 
  enable the IVAD approach for improving the data quality.

A long reach question:  If the we were able to achieve the change noted in red (above),  would you or someone you 
know in the satellite community be willing to convert these research moored buoys to IMMA data format for Release 3.0?

Thanks Chris.  We appreciate the way you use and work on these data.  It helps us understand the user community and
forms the way we go forward.


On Oct 28, 2013, at 12:35 PM, Eric Freeman <Eric.Freeman at noaa.gov> wrote:

> Hi Chris,
> Good to hear from you! Things are going pretty good here, especially for Scott. He's on his annual holiday and expected back in the office next week.
> Hope you are doing well!
> Your questions below cross many different topics the US partners are trying to deal with and NCAR has been the key player in many of these. In the future, please include Steve Worley (cc'd), and between the 3 of us we should be able to answer most of your questions.
> My answers are interleaved below. I may have to defer to Steve on some of these.
> ICOADS 3.0:
> - For the database I'm creating I've made use of the UIDs released recently in IMMA1 format, which I've been referring to as ICOADS 2.5.1 (hopefully correctly!).  I was wondering whether the observations in ICOADS 3.0 will preserve the ICOADS 2.5.1 UIDs?  I appreciate the concept of the UID is one UID per ob consistent across ICOADS updates, but I wasn't sure if those released in ICOADS 2.5.1 were preliminary.
> R2.5.1 is a preliminary product to help us sort this UID and IVAD process out, as well as upgrades to IMMA(1) for R3.0. 
> I'm not certain if these UID's will carry over or if we will start fresh. Steve - can you elaborate on this?
> - Are there any plans for ICOADS 3.0 to make use of high resolution GTMBA data?  This is something that we are being pushed on for satellite SST validation purposes but which is not straightforward to address as, as I'm sure you appreciate, the GTMBA archive is quite fragmented.  It's my understanding that at present ICOADS gathers the high-resolution buoy data but sub-samples to hourly resolution.  I was also wondering what the planned data coverage for ICOADS 3.0 is in terms of TAO/TRITON, PIRATA and RAMA arrays?  If I recall correctly, I think the RAMA data in ICOADS 2.5 is limited.
> The GTMBA data are very fragmented and hard to collect given their scattered structure. Hi frequency data is becoming more common, whether through RVs or buoy observations (e.g. Arctic Shell moored buoys in the Arctic), so it is something we definitely have to deal with. An example is the SAMOS RV data from FSU. They subsample to hourly.
> Steve - are the buoys in WOD13 being added to the Nocn attm subsampled to hourly in cases of high frequency? Plans to grab additional GTMBA data not part of WOD13?
> - I was thinking again about drifting buoys.  Looking at the ICOADS 3.0 preliminary workplan it looks like an update of the ISDM deck is planned, but I've been wondering how I can make the most of my recent investigations into drifting buoy SST data.  I guess a conclusion of this work was that various gross errors are still apparent in delayed mode ICOADS 2.5 drifter data (biases, enhanced noise, buoys out of water etc.) and although I made an attempt to develop QC procedures that address this, this is likely a one-off piece of work and not something we can keep revisiting for future ICOADS releases.  As such, I was wondering how such errors might be minimised in the future.  Two suggestions jump to mind: (i) feeding back our findings to ISDM via ICOADS to see if improvements could be made to the ISDM QC procedures and archive (ii) incorporating the AOML SVP archive held at ISDM into ICOADS; I believe many of these errors are removed by AOML QC (and SVP buoys are approx. 80% of the global array).  This is only something I've been pondering lately but I'd be interested to know what you think.
> It would be great if you could provide any bias adjustments or corrections for the IVAD project where your work has pointed out some of the mentioned obvious issues. We hope you will consider that as we move forward with that product.
> We rely heavily on user feedback and would be willing to work with ISDM and you to provide feedback on their data quality and potential qc enhancements they can put to use in the future.
> The AOML SVP product should also be considered as a new data source for ICOADS, considering the higher quality data and qc procedures  already applied to the data.
> I know my answers are a bit vague, but hopefully Steve can fill in some gaps, and Scott will be back just before our meeting, which looks to be on Wed 6 Nov, and can likely elaborate.
> Cheers,
> Eric
> -- 
> ===========================================
> Eric Freeman
> Marine Observation Analyst
> Team STG, Inc./ERT - Government Contractor
> Ingest and Analysis Branch
> Global Climate Applications Division
> NOAA's National Climatic Data Center
> 151 Patton Avenue, Asheville, NC 28801-5001
> Phone: +1 828.271.4463
> Fax:   +1 828.271.4022
> Eric.Freeman at noaa.gov
> ===========================================

Steven Worley / NCAR
worley at ucar.edu
Wrk: 303.497.1248
Mobile: 720.468.1961

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20131101/177a9bbf/attachment.html 

More information about the icoads_usa mailing list