[Dart-dev] [4865] DART/trunk/observations/text: Revamped the html file quite a bit; added sections with more advice on
nancy at ucar.edu
nancy at ucar.edu
Mon Apr 18 14:56:26 MDT 2011
Revision: 4865
Author: nancy
Date: 2011-04-18 14:56:25 -0600 (Mon, 18 Apr 2011)
Log Message:
-----------
Revamped the html file quite a bit; added sections with more advice on
choosing types, calendars, and in particular setting observation error values.
Changed only a single comment line in the .f90 file; no code changes.
Modified Paths:
--------------
DART/trunk/observations/text/text_to_obs.f90
DART/trunk/observations/text/text_to_obs.html
-------------- next part --------------
Modified: DART/trunk/observations/text/text_to_obs.f90
===================================================================
--- DART/trunk/observations/text/text_to_obs.f90 2011-04-15 23:21:25 UTC (rev 4864)
+++ DART/trunk/observations/text/text_to_obs.f90 2011-04-18 20:56:25 UTC (rev 4865)
@@ -117,7 +117,8 @@
! location: lat, lon, and height in pressure or meters
! time: when the observation was taken
! type: from the DART list of obs types
- ! error: very important - the instrument error plus model error
+ ! error: very important - the instrument error plus representativeness error
+ ! (see html file for more info)
! assume here a line is a type (1/2), location, time, value, obs error
Modified: DART/trunk/observations/text/text_to_obs.html
===================================================================
--- DART/trunk/observations/text/text_to_obs.html 2011-04-15 23:21:25 UTC (rev 4864)
+++ DART/trunk/observations/text/text_to_obs.html 2011-04-18 20:56:25 UTC (rev 4865)
@@ -2,26 +2,41 @@
"http://www.w3.org/TR/html4/strict.dtd">
<HTML>
<HEAD>
-<TITLE>MADIS Data</TITLE>
+<TITLE>program text_to_obs</TITLE>
<link rel="stylesheet" type="text/css" href="../../doc/html/doc.css">
+<link href="../../doc/html/dart.ico" rel="shortcut icon" />
</HEAD>
<BODY>
<A NAME="TOP"></A>
-<center>
-<A HREF="#Overview">OVERVIEW</A> /
+<H1>PROGRAM <em class=program>text_to_obs</em></H1>
+
+<table border=0 summary="" cellpadding=5>
+<tr>
+ <td valign=middle>
+ <img src="../../doc/html/Dartboard9.png" alt="DART project logo" height=70 />
+ </td>
+ <td>
+ <P>Jump to <a href="../../index.html">DART Documentation Main Index</a><br />
+ <small><small>version information for this file: <br />
+ <!-- version tag follows, do not edit -->
+ $Id$</small></small>
+ </P></td>
+</tr>
+</table>
+
<A HREF="#DataSources">DATA SOURCES</A> /
<A HREF="#Programs">PROGRAMS</A> /
-<A HREF="#KnownBugs">KNOWN BUGS</A> /
-<A HREF="#FuturePlans">FUTURE PLANS</A> /
+<A HREF="#Decisions">DECISIONS</A> /
+<A HREF="#References">REFERENCES</A> /
+<A HREF="#Errors">ERRORS</A> /
+<A HREF="#FuturePlans">PLANS</A> /
<A HREF="#Legalese">TERMS OF USE</A>
-</center>
-<H1>Simple Text File to DART Converter</H1>
-<small><P>
-<!-- version tag follows, do not edit -->$Id$
-</P></small>
+<H1>Overview</H1>
+<H4>Text File to DART Converter</H4>
+
<P>
If you have observations in spreadsheet or column format, in text,
with a single line per observation, then the files this directory
@@ -29,20 +44,26 @@
into a format suitable for DART use.
</P>
<P>
-The workflow is you collect the needed information about each
+The workflow is usually:
+<ul>
+<li>read in the needed information about each
observation - location, time, data value, observation type - from
-your data source, and then
-call a series of DART library routines to construct a derived type that
-contains all the observation information. Then another set of DART library
-routines puts them into a time-sorted series. And finally, the last
-routine is a write subroutine that writes out the series to a file.
+a data source (usually a file)</li>
+<li>call a series of DART library routines to construct a derived type that
+contains all the information about a single observation</li>
+<li>call another set of DART library
+routines to put it into a time-sorted series</li>
+<li>repeat the last 2 steps until all observations are processed</li>
+<li>finally, call a write subroutine that writes out the entire series to a file in
+a format that DART can read in</li>
+</ul>
It is not recommended that you try to mimic the ascii file format
by other means; the format is subject to change and the library routines
will continue to be supported even if the physical format changes.
</P>
<P>
If your input data is in some kind of format like netCDF or HDF,
-then one of the other converters (the MADIS ones for netCDF) might
+then one of the other converters (e.g. the MADIS ones for netCDF) might
be a better starting place for adapting code.
</P>
@@ -54,10 +75,12 @@
<H2>DATA SOURCES</H2>
<P>
-This part is up to you. You will need a location, a value,
-a type, and some kind of error estimate. The error estimate
-can be hardcoded in the converter if it's not available in
-the input data.
+This part is up to you.
+For each observation you will need a location, a data value,
+a type, a time, and some kind of error estimate. The error
+estimate can be hardcoded in the converter if they are not
+available in the input data. See below for more details on
+selecting an appropriate error value.
</P>
<!--==================================================================-->
@@ -66,29 +89,33 @@
<HR>
<H2>PROGRAMS</H2>
<P>
-The <em class=file>text_to_obs.f90</em> file is the main converter.
-Look at the source code where it reads the sample data file. You will
-almost certainly need to change the read to match your data. The
-sample code provided reads in each text line into a character buffer
-and then uses reads from that buffer to parse up the data items.
-</P>
-<P>
+The <em class=file>text_to_obs.f90</em> file is the source
+for the main converter program.
+Look at the source code where it reads the example data file. You will
+almost certainly need to change the "read" statement to match your data
+format. The example code
+reads each text line into a character buffer
+and then reads from that buffer to parse up the data items.
+</P> <P>
To compile and test,
go into the work subdirectory and run the <em class=file>quickbuild.csh</em>
script to build the converter and a couple of general purpose utilities.
-<em class=file>advance_time</em> helps compute times, and
+<em class=file>advance_time</em> helps with calendar and time computations,
+and the
<em class=file>obs_sequence_tool</em> manipulates DART observation files
once they have been created.
</P>
<P>
-To add different observation types, look in the
+To change the observation types, look in the
<em class=file>DART/obs_def</em> directory. If you can
find an obs_def_XXX_mod.f90 file with an appropriate set
-of observations, put that name in the <em class=file>input.nml</em>
-namelist file, in the &preprocess_nml namelist, 'input_files'.
-Multiples can be listed. Then run quickbuild.csh again to remake
-the table of supported observation types before trying to recompile
-the source code.
+of observation types, change the 'use' lines in the converter
+source to include those types. Then add that filename in the
+<em class=file>input.nml</em> namelist file
+to the &preprocess_nml namelist, the 'input_files' variable.
+Multiple files can be listed. Then run quickbuild.csh again.
+It remakes the table of supported observation types before
+trying to recompile the source code.
</P>
<P>
An example script for converting batches of files is
@@ -102,6 +129,117 @@
</P>
<!--==================================================================-->
+
+<A NAME="Decisions"></A>
+<HR>
+<H2>DECISIONS YOU MIGHT NEED TO MAKE</H2>
+
+<H4>Time</H4>
+
+<P>
+The basic time type in DART is a pair of integers; one for
+the day number and one for the number of seconds. For some
+of the simple models which aren't direct geophysical models
+time can start at day 0 and increment in any appropriate
+number of seconds or days. For a model of a real-world system
+it is likely to have observations with a year/month/day,
+hour/min/seconds timestamp. There are routines in DART
+to convert back and forth between the (day-number/seconds)
+format and a variety of (year/month/day) calendars.
+See <a href="../../time_manager/time_manager_mod.html#time_type">here</a>
+for more details on how DART stores time information and
+the types of available calendars.
+</P>
+
+<H4>Error</H4>
+
+<P>
+Each observation must specify an associated error.
+There is an error value per individual observation.
+It can be a constant value for all observations
+of that type, or it
+can vary by location, by height, by magnitude of the
+observed value, etc.
+This value is the expected instrument error plus
+the representativeness error of the model.
+The model error includes deficiencies
+in the equations representing the processes of the system
+as well as errors introduced by representing a continuous
+system as a series of discrete points.
+While the instrument error and the representativeness error
+could be specified separately, they each have the same
+impact on the assimilation and can be difficult to determine
+with any real accuracy. For simplicity, in DART (and
+most current assimilation software) they are
+combined and specified as a single value.
+</P><P>
+The instrument error is generally supplied by the instrument
+maker. Sadly, however, it is frequently surprisingly
+difficult to find these values.
+For the representativeness error, a set of artificial
+observations could be generated with the
+<a href="../../perfect_model_obs/perfect_model_obs.html">perfect_model_obs</a>
+program and an assimilation experiment could be run to
+generate an estimate of the error in the model.
+In practice however most people make an educated guess
+on the values of the error and then start with a larger than
+expected value and decrease it based on the results of
+running some test assimilations.
+For these tests the namelist for the
+<a href="../../filter/filter.html#Namelist">outlier threshold</a>
+should be disabled by setting it to -1 (the default value is 3).
+This value controls whether the observation is rejected
+because the observed value is too far from the ensemble mean.
+</P><P>
+If the diagnostics show that the difference between the mean of
+the forward operators and the observed value is consistently
+smaller than the specified observation error, then the error
+is probably too large.
+A too-large error reduces the impact of an
+observation on the state.
+If the specified observation error is too small it is
+likely the observation will be rejected when the outlier
+threshold is enabled, and the observation will not be
+assimilated. It is important
+to look at the output observation sequence files after an assimilation
+to see how many observations were assimilated or rejected, and at
+the error and spread of the ensemble mean compared to the observation
+values.
+</P><P>
+One last recommendation: if possible, the Prior forward operator values
+should be compared against the observations after several assimilation
+cycles. If you plot results using the Posterior values
+it is always possible for the assimilation to overfit the
+observations and look good on the diagnostic plots. But the
+actual test is to then advance the model and look at how the
+forecast of the state compares to the observations.
+</P>
+
+<H4>Types</H4>
+
+<P>
+All observations have to have a specific 'type'.
+There are namelist controls
+to turn on and off the assimilation of observations at run-time
+by type, or to only evaluate the forward operator for an observation
+but have no impact on the state.
+Several of the diagnostics also group observations by
+type to give aggregate statistics after an assimilation.
+Generally types are based on both the observing platform or
+instrument as well as the kind of observation,
+e.g. RADIOSONDE_TEMPERATURE, ARGO_SALINITY, etc.
+Each type is associated with a single underlying generic 'kind',
+which controls what forward operator code is called inside the
+model, e.g. KIND_TEMPERATURE, KIND_DENSITY, etc.
+</P><P>
+See <a href="../../obs_def/obs_def_mod.html">here</a> for more
+details on how to use and add new DART types.
+The DART obs_kind_mod.f90 defines a list of already defined
+observation kinds, and users can either use existing observation
+types in 'obs_def_xxx_mod.f90' files, or define their own.
+</P>
+
+<!--==================================================================-->
<!-- Describe the bugs. -->
<!--==================================================================-->
More information about the Dart-dev
mailing list