[icoads_usa] Need input for IVAD planning

Scott Woodruff scott.d.woodruff at noaa.gov
Sat Feb 1 12:39:44 MST 2014

Hi Shawn,

As we're getting into some nitty gritty discussion of the 
visual/Beaufort wind issues etc., I've cc'd Liz too. Responses 
interleaved below:

On 1/31/14 1:18 PM, Shawn R. Smith wrote:
> Hey all,
> I have been working today with one of my grad students to move forward 
> on the IVAD project, specifically the Beaufort correction. After 
> reviewing some papers (including Lindau's work), I have some questions 
> to raise.
> 1. It is pretty clear that the estimated wind reports have been 
> dropping over the past few decades, but without actually looking at 
> the numbers, I was wondering if anyone had a feel for what year or 
> years we should start with in creating the IVAD attm for Lindau bias 
> adjustment. His paper focused on the 1960-1971 period, so that is one 
> option. It would be nice to have an overlap with the period where Dave 
> can create Tair adjusted IVAD records (in the hope that some records 
> would have both adjustments). The latter would facilitate more 
> interesting "scientific" excersising of the IVAD concept.

I'm not sure how worthwhile this would be, but from 1966 forward you 
would also have the option of using the WMO Pub. 47 metadata in IMMA 
(/Meta-vos/ attm; ref. also 
http://icoads.noaa.gov/e-doc/imma/WMO47IMMA_1966_2007-R2.5.pdf), e.g. 
whether there is an anemometer height provided (HOB) as a cross check. 
Related to this possibility, attached is Fig. 9 from Kent et al. (2007; 

Eventually of course it seems like it would be highly valuable to our 
user community to offer re-calibrated estimated winds much farther back 
(e.g. possible earlier time block starting 1900, when I think steamships 
became predominant?), but I wonder if something like 1960-2007 (end 
R2.5) might be a useful and more manageable initial period.
> Dave - can you remind me over what period of record you can run the 
> Tair correction?
> 2. I have downloaded the R2.5.1i from NCAR, but now think I should go 
> back and get just the 2.5.1, so as not to have to work with the 
> duplicates.
> Dave - have you worked from 2.5.1 or 2.5.1i?
> 3. When trying to identify Beaufort winds in the IMMA, we plan to 
> extract records based on the status of the wind indicator (WI). From 
> what I can tell indicators 0, 2, 3, 5, and 6 are all "estimated" winds 
> (implying Beaufort), though only 5 is actually labeled Beaufort. I 
> assume 5 is a rather new useage. Are we "safe" to assume any estimated 
> wind is derived from a Beaufort approach (whether a conversion by 
> ICOADS from an actual Beaufort wind # or a wind value extracted from a 
> Beaufort scale as was WMO convention in the post WWII years)?
While all the estimated winds should be visual observations, WI=6 (as 
discussed under WI in 
http://icoads.noaa.gov/e-doc/imma/R2.5-imma_short.pdf) is a bit 
uncertain, because of unresolvable issues in how the original WI 
information was stored (e.g. in NCDC's old TDF-11 format). As you may 
know, this problem is also discussed some in the literature (e.g. Kent 
et al. 2007).

Correct, WI=5 was a relatively new configuration set (during 
"preconditioning") for some decks from R2.0/2.1, when we concluded from 
the documentation that the original data were Beaufort force codes. If 
you're interested, the PS at the end here provides a fairly complicated 
example of such a preconditioning rule for one deck.

At present, ICOADS has only done a fairly limited number of conversions 
from original Beaufort codes to store a m/s value (specifically a WMO 
Code 1100 "old" midpoint, ref. Release 1, Supp. K, Table K5-5) , 
probably mostly for newly digitized/available sources.

More info in case useful on the post-WWII coding situation (as you 
mentioned above): When the GTS SHIP code came into effect (by 1949 
according to: 
http://goos.kishou.go.jp/ws/ETMC/code_task/history/SHIP.html) observers 
could report knots (and importantly weren't constrained to Beaufort mid 
points). Similarly when the punched card (IMMPC) precursor to the IMMT 
keyed logbook format was introduced in the 1950s (e.g. 1957 version here 
http://goos.kishou.go.jp/ws/ETMC/code_task/history/IMMT.html has 
"Beaufort force" but then "knots" are coded). Then at some point both 
GTS/IMMT codes evolved so that whole m/s could be reported instead of 
whole knots (as flagged by a wind speed indicator).

Also in case helpful, the second attachment provides some 1992 notes 
that Joe Elms put together on several older historical ICOADS decks (to 
dig deeper, see also original card deck documentation at: 
http://icoads.noaa.gov/reclaim/us.html), as to the digital data 
availability and original Beaufort code reporting procedures (which I 
may have sent you before, but maybe worth another look now).

> 4. Similarly, we will be extracting ship only records by looking at 
> the platform type (PT). Looks like PT = 0, 1, 2, 3, 4, and 5 are 
> ships. Any gotcha's with this assumption?

Looks good to me, except PT=missing would be one other possibility, and 
depending on the time period/deck one might conclude a given report with 
PT=missing is most likely to be from a ship.
> 5. Also another history lesson for me. Am I correct that wind speed 
> (W) has always been stored in m/s in ICOADS? Lindau's manuscript 
> implies that it was stored in kts, but looking at the original ICOADS1 
> documentation indicates that m/s was always used.

Correct, ICOADS has always stored wind speeds in m/s. As noted above, 
the current SHIP (FM13)/IMMT codes allows VOS to report whole knots or 
whole m/s. (Looking forward, I'm not up to date with how wind speeds are 
handled in the latest incarnation of the VOS BUFR template, but Dave 
could comment.)
> Finally, next week I will take a look at the IVAD attm structure as it 
> stands now. Unless Dave has any concerns, I think we should lock this 
> down as the version for all the final prototyping. Hopefully that will 
> allow Steve to move forward with mapping some IVAD records into the 
> database. More next week.
If you think this is workable, I'd like to suggest that we transition to 
the revised /Ivad/ attm structure as agreed at our Nov. 2013 Boulder 
IVAD meeting.

Sorry that this has taken much longer than anticipated, but I'm reaching 
closure (hopefully by the end of next week) on a full new version (v15) 
of the Proposed IMMA Revisions document, incorporating many decisions we 
took at the November meeting. Meanwhile, attached as a preview are just 
Tables C96 (/Ivad/ attm) and C96a (/VQC/ flag configuration, so as 
agreed we are dropping the IMMT flag scheme for /VQC/) incorporating 
changes agreed last Nov.

Also, Sandy recently updated the {rdimma1} software 
(http://icoads.noaa.gov/software/rdimma1) to handle I/O for the 
finalized /Nocn/ attm (e.g. for WOD13), and, while I'll need to check 
with her on the logistics/timing, I believe she was also on the verge of 
enabling the /Ivad/ (and /Error)/ attms in their currently documented 
(ref. http://icoads.noaa.gov/ivad/IMMA-Rev-v14.pdf) form, so my 
suspicion is that we will have both the documentation (note: pending 
checking proposed new field abbreviations for uniqueness etc.) and 
software for the revised /Ivad/ attm as attached locked down fairly soon.

Sorry for such a long and complicated e-mail, but I hope it's helpful 
for getting moving on the scientific IVAD demonstration. -Scott

PS: Example Release 2.0 preconditioning rule for deck 187 (from 
Deck 187: Japanese Whaling Fleet
        Rules:  If both wind speed (W) and direction (D) are missing 
(and not
        in the error attachment), and if 12-point wind direction is 
"12", and
        Beaufort force falls in the range 0-9 or is "-" or "+" (fields 
in the
        supplemental attachment corresponding to original TDF-11 
positions 114-
        115 and 116, respectively).  If these conditions are all true, 
make the
        following settings, ignoring any existing settings of the wind 
        indicator (DI) and wind speed indicator (WI):
             i) DI to missing.
             ii) D to 360 degrees.
             iii) WI to 5 (Beaufort force).
             iv) Convert Beaufort force to W in meters per second using the
             midpoint of the "old" Beaufort equivalence scale, where 0-9 are
             mapped to meters per second according to Release 1, Table K5-5,
             "-" (equivalent to Beaufort force 10) is mapped to 26.8, 
and "+"
             (equivalent to Beaufort force 11 or higher) to 30.9.

        Background:  In conversion from the original card format into 
        errors appear to have been made in failing to convert any direction
        code 36 (north in the 36-point scale) and its associated wind speed.
        However, a 12-point wind direction code and Beaufort force 
usually seem
        to be available in the supplemental attachment to provide a 
        [NOTE: This correction is approximate because D and W in TDF-11 were
        derived from separate fields, when extant, that appeared only in the
        original card format (36-point direction and speed in knots); the
        relationship between the two different sets of D and W fields in the
        original card format is unclear.  DI is set to missing because we
        currently lack a DI value for indication of the 12-point scale.  In
        addition, the 12-point wind direction code of "12" corresponds to a
        range of 335-4 degrees, whose midpoint is not 360 degrees. Moreover,
        Beaufort value "+" actually signifies force 11 or greater (up to 
        according to the original card deck reference manual; usage of the
        midpoint for force 11 is thus definitely an approximation. Assuming
        that the original card deck reference manual is correct, NCDC (1968)
        incorrectly documents the Beaufort value "+" as signifying force 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140201/c5b30cf0/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Kent-et-al-2007-FIG9.jpeg
Type: image/jpeg
Size: 100429 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140201/c5b30cf0/attachment-0001.jpeg 
-------------- next part --------------
>From  sdw (=SDW)
Subject: Beaufort documentation
To: moninger at fsl.noaa.gov,
Resent-Message-ID: <920420131115B07.AYEB at NOAA.CS2> (UMass-Mailer 4.04)
Resent-Date: Mon, 20 Apr 92  13:11:21 MST
Resent-From: sdw (=SDW)
Resent-To: mailco
Status: RO
Content-Length: 9390

Bill,  Joe Elms and I have discussed the attached documentation--perhaps it
should go into Metalog now.  -Scott
Return-path: <jelms at gclsun01.ncdc.noaa.gov>
Received: from BNOS.BLDRDOC.GOV by NOAA.CS2 with BSMTP;
          17 Dec 91 12:16:16 MST
Received: from ibmser01.ncdc.noaa.gov by BNOS.BLDRDOC.GOV
 with SMTP; 17 Dec 91 12:15:50 MST
Received: from gclsun01.ncdc.noaa.gov by ibmser01.ncdc.noaa.gov (AIX 3.1/UCB 5.61/4.03)
          id AA08844; Tue, 17 Dec 91 14:16:48 -0500
Received: by gclsun01.ncdc.noaa.gov (4.1/SMI-4.1)
 id AA00240; Tue, 17 Dec 91 14:12:20 EST
Date: Tue, 17 Dec 91 14:12:20 EST
From: jelms at gclsun01.ncdc.noaa.gov (Joe Elms)
Message-Id: <9112171912.AA00240 at gclsun01.ncdc.noaa.gov>
To:   sdw at bnos.bldrdoc.gov
Subject: BF documentaton
Attached is the documentation I put together on the various historical
decks and their use of the Beaufort Force code. Please review and let me
have your comments. I can make the needed change and we can discuss what
we want to do with it (in-house documentation, included in reference manual
or improved to be used as a handout at the workshop).
              Beaufort Force Documentation
The historical "Card Deck Reference Manuals" were reviewed to
determine which source decks had wind speeds originally observed
in the Beaufort Force Code. Many fit this criteria, although the
accompanying wind speed scales (knots,mph, etc.) were not always
consistent. A few of the source decks have a mixture of reporting
methods (BF and Knots) depending on period of record or ship
origin.  This information is outlined in the following tables.
Table 1.  Original observations reported in the Beaufort Force
Deck     Period of Record      BF Scale
184       4/1953 - 6/1956         A
194       1856 - 1950             A*
192       1859 - 1939             A
118       1933 - 1953             A
193       1854 - 1938             A
188       1920 - 1945             B
A* - the original card deck contained the Beaufort Force Number
as observed with the BF being converted to knots during the
conversion to the magnetic tape format according to scale A.
However, the comparable wind speed table associated with each
force number was originally in miles per hour and could have
influenced the observers reported value differently than if the
comparable codes had been in knots or word descriptions, as was
the general practice, for the other decks using the scale A.
                 SCALE A
BF#           Range           Conversion Value
      A (knots)   A* (mph)
0      calm           <1          0
1      1 - 3        1 - 3         2
2      4 - 6        4 - 7         5
3      7 -10        8 -12         9
4      11-16        13-18         13
5      17-21        19-24         18
6      22-27        25-31         24
7      28-33        32-38         30
8      34-40        39-46         37
9      41-47        47-54         44
10     48-55        55-63         52
11     56-63        64-72         60
12     >63          >72           68
BF#    As Keyed
0        0
1        2
2        5
3        9
4        14 @
5        19 @            Note: @ indicates those values keyed in
6        25 @                  knots for  BF # that differ from
7        31 @                  those used in Scale A.
8        37
9        44
10       51 @
11       58 @
12       66 @
Table 2.  Original observations reported in Knots
Deck     Period of Record
902        1956 - 1960     referred to as DK 184 extension
119        1953 - 1960
110        1945 - 6/1951
195        1942 - 1945
189        1939 - 1955     mixed 1949 & 1955 codes
281        1920 - 1945
186        1950 - 1960
196        1949 - 1953     knots converted to BF in supplemental
Table 3.  Original observations reported in meters per second
Deck     Period of Record
185        6/1957 - 12/1958        converted to knots
                                   (x 1.94254 & rounded to
                                    nearest whole value)
Table 4.  Original observations reported in mixed units
Deck     Period of Record
116        1949-1963            When converting from the card
                                format to that for tape, if
                                columns 25-26 (wind speed in
                                knots) was missing then column
                                27 (BF #) was converted to
                                knots and placed in the missing
                                wind field.
197        1860 - 1956          Those observed in knots were
                                converted to BF and those
                                observed in BF were converted to
                                knots. It is now impossible to
                                clearly establish the original
                                units. Some original observations
                                were even reported in the Ben
                                Nevis code, which were
                                subsequently converted per the
                                following table.
               Wind Conversion from Ben Nevis Observatory Code to
                           Beaufort Force and Knots
             Ben      BF      Knots
              0       0         0
              0-1     1         1-3
              1       2         4-6
              2       3         7-10
              2-3     4        11-16
              3       5        17-21
              3-4     6        22-27
              4       7        28-33
              5       8        34-40
              6       9        41-47
              7       10       48-55
              8       11       56-63
              9       12       64-71
              10      12       72-80
              11      12       81-89
              12      12       90
187     11/1946 -3/1956         In most DK 187 observations the
        (Nov-Mar only)          wind speed appears in both BF and
                                knots. Examination of the data
                                revealed that the conversions
                                were often from knots to BF
                                because for a given BF the full
                                range of possible speeds in knots
                                appear. It is not clear if the
                                reverse is true; that any of the
                                original reports were in BF and
                                converted to knots. However,
                                there was a conversion error made
                                that failed to convert any
                                directional code 36's
                                (north in the 36 pt scale) and
                                its associated wind speed.
                                Information does appear in the 12
                                point direction scale and BF
                                fields for the missing values.
                                This will allow corrections to be
                                made, but resulting in a blended
                                mixture of BF and Knot
                                conversions. It is probable that
                                this mixture already exists.
NOTE: Another problem with Deck 187 is that many of the original
      winds were reported using a 16 point scale, but these were
      converted to a 36 point scale before being keyed. This
      clearly biases the directions and with no indicators being
      established at the time of keying, the data cannot be
      corrected except through statistical schemes. There is no
      documented evidence as to whether any of the original
      observations were reported in the 12 point scale or if it
      was established only for summary purposes. Two different
      conversion schemes appear to have been used in converting
      the 36/16 point wind directions to the 12 point
      directional codes. All the latter years (1951 and later)
      were converted using the same scheme while the early years
      appear to have been converted using two different methods.
      The original card deck documentation (deck 187 - Japanese
      Whaling Ships) states that the early period (deck 198)
      contained erroneous data and that it was corrected and five
      additional years added when the new deck number (187) was
      established (this could account for these problems if all
      the early period errors were not corrected). No
      documentation could be located for deck 198.
Table 5.  A mixture of various source decks
901    These were extreme values from various decks (110,116,
       118,119,184,189,192,193,194,195, & 281) that were
       eliminated during the 1950's marine atlas project,
       but retained on FOSDIC (Film Optical Sensing Device
       Input to Computer). These were selected off FOSDIC and
       added back into the marine data base as DK 902 since the
       original source I.D. had been lost. The number of 902
       observations is very small.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMMA-Rev-v15_WORKING-TablesC96-C96a.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 48767 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/icoads_usa/attachments/20140201/c5b30cf0/attachment-0001.bin 

More information about the icoads_usa mailing list