[Dart-dev] [4331] DART/trunk/observations: An example of a simple text-to-obs_seq file converter.
nancy at ucar.edu
nancy at ucar.edu
Tue Mar 30 09:11:11 MDT 2010
Revision: 4331
Author: nancy
Date: 2010-03-30 09:11:11 -0600 (Tue, 30 Mar 2010)
Log Message:
-----------
An example of a simple text-to-obs_seq file converter.
Intended to be used as a template for users with text
data. Users with data already in netCDF or HDF would do
better to start with the MADIS converters as a template.
Added Paths:
-----------
DART/trunk/observations/text/
DART/trunk/observations/text/data/
DART/trunk/observations/text/data/README
DART/trunk/observations/text/data/text_input_file
DART/trunk/observations/text/shell_scripts/
DART/trunk/observations/text/shell_scripts/hour_time_loop.csh
DART/trunk/observations/text/shell_scripts/input.nml
DART/trunk/observations/text/shell_scripts/input.nml.template
DART/trunk/observations/text/text_to_obs.f90
DART/trunk/observations/text/text_to_obs.html
DART/trunk/observations/text/work/
DART/trunk/observations/text/work/input.nml
DART/trunk/observations/text/work/mkmf_advance_time
DART/trunk/observations/text/work/mkmf_obs_sequence_tool
DART/trunk/observations/text/work/mkmf_preprocess
DART/trunk/observations/text/work/mkmf_text_to_obs
DART/trunk/observations/text/work/path_names_advance_time
DART/trunk/observations/text/work/path_names_obs_sequence_tool
DART/trunk/observations/text/work/path_names_preprocess
DART/trunk/observations/text/work/path_names_text_to_obs
DART/trunk/observations/text/work/quickbuild.csh
-------------- next part --------------
Added: DART/trunk/observations/text/data/README
===================================================================
--- DART/trunk/observations/text/data/README (rev 0)
+++ DART/trunk/observations/text/data/README 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,31 @@
+This directory contains a very simple text input file
+to use as an example of reading in text data. The observations
+are one per line, and the items on each line are space separated;
+fixed width columns work well, too.
+
+In this simple example, the first item on each line is an observation
+type code (1=temperature, 2=wind).
+
+The next two are latitude/longitude locations.
+
+The next is the vertical elevation. For temperature it is
+in meters, for winds it is in pressure. Observations can
+use either type of vertical units.
+
+Then the time of the observation - year, month, day, hour, minute,
+second.
+
+The last values are the actual observation values and an error
+estimate. For temperature it is degrees kelvin, +/- 2 degrees.
+For wind, it is speed in m/s, direction in degrees, and an error
+estimate of 4 m/s.
+
+This is just an example that matches the read code in the
+converter - any variation of this is easy to support, with
+changes to the read lines to get the right information in
+the right places. If there is no error estimate (a common
+problem), the converter can supply a reasonable fixed value.
+
+ 1 -45 270 10 2008 1 30 12 0 0 273 2
+ 2 30 300 998 2008 1 31 6 30 0 10 355 4
+
Property changes on: DART/trunk/observations/text/data/README
___________________________________________________________________
Added: svn:mime-type
+ text/plain
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/data/text_input_file
===================================================================
--- DART/trunk/observations/text/data/text_input_file (rev 0)
+++ DART/trunk/observations/text/data/text_input_file 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,2 @@
+ 1 -45 270 10 2008 1 30 12 0 0 273 2
+ 2 30 300 998 2008 1 31 6 30 0 10 355 4
Added: DART/trunk/observations/text/shell_scripts/hour_time_loop.csh
===================================================================
--- DART/trunk/observations/text/shell_scripts/hour_time_loop.csh (rev 0)
+++ DART/trunk/observations/text/shell_scripts/hour_time_loop.csh 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,158 @@
+#!/bin/csh
+#
+# DART software - Copyright \xA9 2004 - 2010 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# $Id$
+#
+# this is a template for a shell script that can loop
+# over multiple hours and can roll over day, month, and
+# even year boundaries. see the section inside the loop
+# for the 'your code goes here' part. look for the string ADDME.
+# this script requires the executable 'advance_time' to be
+# built and exist in the current directory, and advance_time
+# requires an input.nml namelist file.
+
+
+# set this to true if you're planning to pass the start & end times
+# in as command line args. set it to false if you're planning to set
+# the times by editing this file.
+
+set command_line_args = false
+
+# set the first and last times. can roll over day, month and year boundaries.
+# hours go from 0 to 23; days from 1 to 31, months from 1 to 12.
+
+if ($command_line_args == 'true') then
+ if ($#argv != 8) then
+ echo usage: $0 start_year start_month start_day start_hour end_year end_month end_day end_hour
+ exit 1
+ endif
+ set start_year=$argv[1]
+ set start_month=$argv[2]
+ set start_day=$argv[3]
+ set start_hour=$argv[4]
+
+ set end_year=$argv[5]
+ set end_month=$argv[6]
+ set end_day=$argv[7]
+ set end_hour=$argv[8]
+else
+ set start_year=2006
+ set start_month=10
+ set start_day=31
+ set start_hour=18
+
+ set end_year=2006
+ set end_month=11
+ set end_day=1
+ set end_hour=12
+endif
+
+# <ADDME> put more stuff here if you have user settable options
+
+# end of things you should have to set in this script
+
+# convert the start and stop times to gregorian days, so we can
+# compute total number of days including rolling over month and
+# year boundaries. make sure all values have leading 0s if they
+# are < 10. do the end time first so we can use the same values
+# to set the initial day while we are doing the total day calc.
+
+# the output of advance time with the -g input is:
+# gregorian_day_number seconds
+# use $var[1] to return just the day number
+
+set mon2=`printf %02d $end_month`
+set day2=`printf %02d $end_day`
+set hr2=`printf %02d $end_hour`
+set end_t=(`echo ${end_year}${mon2}${day2}${hr2} 0 -g | ./advance_time`)
+
+set mon2=`printf %02d $start_month`
+set day2=`printf %02d $start_day`
+set hr2=`printf %02d $start_hour`
+set start_t=(`echo ${start_year}${mon2}${day2}${hr2} 0 -g | ./advance_time`)
+
+# the output of this call is a string YYYYMMDDHH
+# see below for help in how to easily parse this up into words
+set curhr=`echo ${start_year}${mon2}${day2}${hr2} 0 | ./advance_time`
+
+# how many total hours are going to be processed (for the loop counter)
+# note that the parens below are necessary; otherwise the computation
+# does total = end - (start+1), or total = end - start - 1, which is
+# not how elementary math is supposed to work.
+if ( $start_t[2] > $end_t[2]) then
+ @ end_t[2] += 86400
+ @ end_t[1] -= 1
+endif
+@ totaldays = ( $end_t[1] - $start_t[1] )
+@ totalsecs = ( $end_t[2] - $start_t[2] )
+@ totalhrs = ($totaldays * 24) + ($totalsecs / 3600) + 1
+echo days, secs = hrs: $totaldays, $totalsecs = $totalhrs
+
+# loop over each hour
+set h=1
+while ( $h <= $totalhrs )
+
+ # parse out the parts from a string which is YYYYMMDDHH
+ # use cut with the byte option to pull out columns 1-4, 5-6, 7-8, 9-10
+ set year=`echo $curhr | cut -b1-4`
+ set month=`echo $curhr | cut -b5-6`
+ set day=`echo $curhr | cut -b7-8`
+ set hour=`echo $curhr | cut -b9-10`
+
+ # compute the equivalent gregorian day here.
+ set g=(`echo ${year}${month}${day}${hour} 0 -g | ./advance_time`)
+ set gregday=$g[1]
+ set gregsec=$g[2]
+
+ # status/debug - comment in or out as desired.
+ echo starting processing for ${year} ${month} ${day} ${hour}
+ echo which is gregorian day: $gregday, $gregsec
+
+
+ # <ADDME> your code goes here.
+ # use $year, $month, $day, $hour, and $gregday, $gregsec as needed.
+
+
+ # advance the hour; the output is YYYYMMDDHH
+ set curhr=`echo ${year}${month}${day}${hour} +1h | ./advance_time`
+
+ # advance the loop counter
+ @ h += 1
+
+end
+
+exit 0
+
+
+#%# # example of using sed and lists of obs files to automate
+#%# # calling the obs_sequence_tool to split or combine obs_seq files:
+#%#
+#%# # put a list of filenames into 'obstemp' somehow
+#%#
+#%# # remove duplicate filenames
+#%# sort obstemp | uniq > infilelist
+#%# echo 'using input files:'
+#%# cat infilelist
+#%#
+#%# # if the start and stop times are in gregorian format,
+#%# # in $start and $stop, use sed to set the input.nml
+#%# sed -e "s/BDAY/$start[1]/" \
+#%# -e "s/BSEC/$start[2]/" \
+#%# -e "s/ASEC/$stop[2]/" \
+#%# -e "s/ASEC/$stop[2]/" input.nml.template >! input.nml
+#%#
+#%# # run obs_seq_tool
+#%# ./obs_sequence_tool
+#%#
+#%# # move the output someplace
+#%# mv obs_seq.combined obs_seq.$curhr
+#%#
+
+# <next few lines under version control, do not edit>
+# $URL$
+# $Revision$
+# $Date$
+
Property changes on: DART/trunk/observations/text/shell_scripts/hour_time_loop.csh
___________________________________________________________________
Added: svn:executable
+ *
Added: svn:mime-type
+ text/plain
Added: svn:keywords
+ Date Rev Author HeadURL Id
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/shell_scripts/input.nml
===================================================================
--- DART/trunk/observations/text/shell_scripts/input.nml (rev 0)
+++ DART/trunk/observations/text/shell_scripts/input.nml 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,35 @@
+
+&preprocess_nml
+ input_obs_kind_mod_file = '../../../obs_kind/DEFAULT_obs_kind_mod.F90',
+ output_obs_kind_mod_file = '../../../obs_kind/obs_kind_mod.f90',
+ input_obs_def_mod_file = '../../../obs_def/DEFAULT_obs_def_mod.F90',
+ output_obs_def_mod_file = '../../../obs_def/obs_def_mod.f90',
+ input_files = '../../../obs_def/obs_def_eval_mod.f90',
+ /
+
+&obs_kind_nml
+ /
+
+&location_nml
+ /
+
+&utilities_nml
+ module_details = .false.
+ /
+
+&obs_sequence_nml
+ write_binary_obs_sequence = .false.
+ /
+
+
+&obs_sequence_tool_nml
+ filename_seq = '',
+ filename_out = 'obs_seq.combined',
+ filename_seq_list = 'infilelist',
+ first_obs_days = -1,
+ first_obs_seconds = -1,
+ last_obs_days = -1,
+ last_obs_seconds = -1,
+ gregorian_cal = .true.,
+/
+
Property changes on: DART/trunk/observations/text/shell_scripts/input.nml
___________________________________________________________________
Added: svn:mime-type
+ text/plain
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/shell_scripts/input.nml.template
===================================================================
--- DART/trunk/observations/text/shell_scripts/input.nml.template (rev 0)
+++ DART/trunk/observations/text/shell_scripts/input.nml.template 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,35 @@
+
+&preprocess_nml
+ input_obs_kind_mod_file = '../../../obs_kind/DEFAULT_obs_kind_mod.F90',
+ output_obs_kind_mod_file = '../../../obs_kind/obs_kind_mod.f90',
+ input_obs_def_mod_file = '../../../obs_def/DEFAULT_obs_def_mod.F90',
+ output_obs_def_mod_file = '../../../obs_def/obs_def_mod.f90',
+ input_files = '../../../obs_def/obs_def_eval_mod.f90',
+ /
+
+&obs_kind_nml
+ /
+
+&location_nml
+ /
+
+&utilities_nml
+ module_details = .false.
+ /
+
+&obs_sequence_nml
+ write_binary_obs_sequence = .false.
+ /
+
+
+&obs_sequence_tool_nml
+ filename_seq = '',
+ filename_out = 'obs_seq.combined',
+ filename_seq_list = 'infilelist',
+ first_obs_days = BDAY,
+ first_obs_seconds = BSEC,
+ last_obs_days = ADAY,
+ last_obs_seconds = ASEC,
+ gregorian_cal = .true.,
+/
+
Added: DART/trunk/observations/text/text_to_obs.f90
===================================================================
--- DART/trunk/observations/text/text_to_obs.f90 (rev 0)
+++ DART/trunk/observations/text/text_to_obs.f90 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,346 @@
+! DART software - Copyright \xA9 2004 - 2010 UCAR. This open source software is
+! provided by UCAR, "as is", without charge, subject to all terms of use at
+! http://www.image.ucar.edu/DAReS/DART/DART_download
+
+program text_to_obs
+
+! <next few lines under version control, do not edit>
+! $URL$
+! $Id$
+! $Revision$
+! $Date$
+
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!
+! text_to_obs - a program that only needs minor customization to read
+! in a text-based dataset - either white-space separated values or
+! fixed-width column data.
+!
+! created 29 Mar 2010 nancy collins NCAR/IMAGe
+!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+use types_mod, only : r8, PI, DEG2RAD
+use utilities_mod, only : initialize_utilities, finalize_utilities, &
+ open_file, close_file
+use time_manager_mod, only : time_type, set_calendar_type, set_date, &
+ operator(>=), increment_time, get_time, &
+ operator(-), GREGORIAN, operator(+), print_date
+use location_mod, only : VERTISHEIGHT, VERTISPRESSURE
+use obs_sequence_mod, only : obs_sequence_type, obs_type, read_obs_seq, &
+ static_init_obs_sequence, init_obs, write_obs_seq, &
+ init_obs_sequence, get_num_obs, &
+ set_copy_meta_data, set_qc_meta_data
+use obs_kind_mod, only : EVAL_U_WIND_COMPONENT, EVAL_V_WIND_COMPONENT, &
+ EVAL_TEMPERATURE
+
+implicit none
+
+character(len=64), parameter :: text_input_file = 'textdata.input'
+character(len=64), parameter :: obs_out_file = 'obs_seq.out'
+
+logical, parameter :: debug = .false. ! set to .true. to print info
+
+character (len=129) :: input_line
+
+integer :: n, i, oday, osec, rcio, iunit, otype
+integer :: year, month, day, hour, minute, second
+integer :: num_copies, num_qc, max_obs
+
+logical :: file_exist, first_obs
+
+real(r8) :: temp, terr, qc, wdir, wspeed, werr
+real(r8) :: lat, lon, vert, uwnd, uerr, vwnd, verr
+
+type(obs_sequence_type) :: obs_seq
+type(obs_type) :: obs, prev_obs
+type(time_type) :: comp_day0, time_obs, prev_time
+
+! start of executable code
+
+call initialize_utilities('text_to_obs')
+
+! time setup
+call set_calendar_type(GREGORIAN)
+
+!! some times are supplied as number of seconds since some reference
+!! date. This is an example of how to support that.
+!! put the reference date into DART format
+!comp_day0 = set_date(1970, 1, 1, 0, 0, 0)
+
+! open input text file
+
+iunit = open_file(text_input_file, 'formatted', 'read')
+if (debug) print *, 'opened input file ' // trim(text_input_file)
+
+
+! each observation in this series will have a single observation value
+! and a quality control flag. the max possible number of obs needs to
+! be specified but it will only write out the actual number created.
+max_obs = 100000
+num_copies = 1
+num_qc = 1
+
+! call the initialization code, and initialize two empty observation types
+call static_init_obs_sequence()
+call init_obs(obs, num_copies, num_qc)
+call init_obs(prev_obs, num_copies, num_qc)
+first_obs = .true.
+
+! create a new, empty obs_seq file. you must give a max limit
+! on number of obs. increase the size if too small.
+call init_obs_sequence(obs_seq, num_copies, num_qc, max_obs)
+
+! the first one needs to contain the string 'observation' and the
+! second needs the string 'QC'.
+call set_copy_meta_data(obs_seq, 1, 'observation')
+call set_qc_meta_data(obs_seq, 1, 'Data QC')
+
+! if you want to append to existing files (e.g. you have a lot of
+! small text files you want to combine), you can do it this way,
+! or you can use the obs_sequence_tool to merge a list of files
+! once they are in DART obs_seq format.
+
+! ! existing file found, append to it
+! inquire(file=obs_out_file, exist=file_exist)
+! if ( file_exist ) then
+! call read_obs_seq(obs_out_file, 0, 0, max_obs, obs_seq)
+! endif
+
+! Set the DART data quality control. 0 is good data.
+! increasingly larger QC values are more questionable quality data.
+qc = 0.0_r8
+
+obsloop: do ! no end limit - have the loop break when input ends
+
+ ! read in a line from the text file. What you need to create an obs:
+ ! location: lat, lon, and height in pressure or meters
+ ! time: when the observation was taken
+ ! type: from the DART list of obs types
+ ! error: very important - the instrument error plus model error
+
+ ! assume here a line is a type (1/2), location, time, value, obs error
+
+ ! read in entire text line into a buffer
+ read(iunit, "(A)", iostat=rcio) input_line
+ if (rcio /= 0) then
+ if (debug) print *, 'got bad read code from input file, rcio = ', rcio
+ exit obsloop
+ endif
+
+ ! pull off the first 2 columns as an integer, to decode the type
+ read(input_line, "(I2)", iostat=rcio) otype
+ if (rcio /= 0) then
+ if (debug) print *, 'got bad read code trying to get obs type, rcio = ', rcio
+ exit obsloop
+ endif
+
+ if (debug) print *, 'next observation type = ', otype
+
+ ! for this example, assume there is an obs type, where otype=1 is
+ ! a temperature measured in height, and if otype=2, there's a wind
+ ! speed and direction.
+
+ if (otype == 1) then
+ read(input_line(3:129), *, iostat=rcio) lat, lon, vert, &
+ year, month, day, hour, minute, second, &
+ temp, terr
+ if (rcio /= 0) then
+ if (debug) print *, 'got bad read code getting rest of temp obs, rcio = ', rcio
+ exit obsloop
+ endif
+ else
+ read(input_line(3:129), *, iostat=rcio) lat, lon, vert, &
+ year, month, day, hour, minute, second, &
+ wspeed, wdir, werr
+ if (rcio /= 0) then
+ if (debug) print *, 'got bad read code getting rest of wind obs, rcio = ', rcio
+ exit obsloop
+ endif
+ endif
+
+ if (debug) print *, 'next observation located at lat, lon = ', lat, lon
+
+ ! check the lat/lon values to see if they are ok
+ if ( lat > 90.0_r8 .or. lat < -90.0_r8 ) cycle obsloop
+ if ( lon < 0.0_r8 .or. lon > 360.0_r8 ) cycle obsloop
+
+
+ ! if lon comes in between -180 and 180, use these lines instead:
+ !if ( lon > 180.0_r8 .or. lon < -180.0_r8 ) cycle obsloop
+ !if ( lon < 0.0_r8 ) lon = lon + 360.0_r8 ! changes into 0-360
+
+ ! put date into a dart time format
+ time_obs = set_date(year, month, day, hour, minute, second)
+
+ if (debug) call print_date(time_obs, 'next obs time is')
+
+ !! if time is given in seconds since 1/1/1970, here's how to add it.
+ !time_obs = comp_day0 + time_obs
+
+ ! extract time of observation into gregorian day, sec.
+ call get_time(time_obs, osec, oday)
+
+ ! this example assumes there is an obs type, where otype=1 is
+ ! a temperature measured in height, and if otype=2, there's a wind
+ ! speed and direction and height is pressure. any kind of observation
+ ! can use any of the vertical types; this is just an example.
+
+ if (otype == 1) then
+
+ ! height is in meters (gph)
+
+ ! make an obs derived type, and then add it to the sequence
+ call create_3d_obs(lat, lon, vert, VERTISHEIGHT, temp, &
+ EVAL_TEMPERATURE, terr, oday, osec, qc, obs)
+ call add_obs_to_seq(obs_seq, obs, time_obs, prev_obs, prev_time, first_obs)
+
+ if (debug) print *, 'added temperature obs to output seq'
+ else
+
+ ! DART usually assimilates wind as 2 separate U and V components
+ ! instead of trying to assimilate a vector of speed and direction.
+ ! so convert a wind speed & direction into the U and V components
+ ! and create 2 obs for it. assume vert is in mb or hectopascals,
+ ! convert to pascals. DART does assume all pressures are in pascals.
+ uwnd = sin(wdir * DEG2RAD) * wspeed
+ vwnd = cos(wdir * DEG2RAD) * wspeed
+ uerr = sin(wdir * DEG2RAD) * werr
+ verr = cos(wdir * DEG2RAD) * werr
+
+ ! convert hectopascals to pascals.
+ vert = vert * 100.0_r8
+
+ call create_3d_obs(lat, lon, vert, VERTISPRESSURE, uwnd, &
+ EVAL_U_WIND_COMPONENT, uerr, oday, osec, qc, obs)
+ call add_obs_to_seq(obs_seq, obs, time_obs, prev_obs, prev_time, first_obs)
+
+ call create_3d_obs(lat, lon, vert, VERTISPRESSURE, vwnd, &
+ EVAL_V_WIND_COMPONENT, verr, oday, osec, qc, obs)
+ call add_obs_to_seq(obs_seq, obs, time_obs, prev_obs, prev_time, first_obs)
+
+ if (debug) print *, 'added 2 wind obs to output seq'
+ endif
+
+end do obsloop
+
+! if we added any obs to the sequence, write it out to a file now.
+if ( get_num_obs(obs_seq) > 0 ) then
+ if (debug) print *, 'writing obs_seq, obs_count = ', get_num_obs(obs_seq)
+ call write_obs_seq(obs_seq, obs_out_file)
+endif
+
+! end of main program
+call finalize_utilities()
+
+contains
+
+
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!
+! create_3d_obs - subroutine that is used to create an observation
+! type from observation data.
+!
+! NOTE: assumes the code is using the threed_sphere locations module,
+! that the observation has a single data value and a single
+! qc value, and that this obs type has no additional required
+! data (e.g. gps and radar obs need additional data per obs)
+!
+! lat - latitude of observation
+! lon - longitude of observation
+! vval - vertical coordinate
+! vkind - kind of vertical coordinate (pressure, level, etc)
+! obsv - observation value
+! okind - observation kind
+! oerr - observation error
+! day - gregorian day
+! sec - gregorian second
+! qc - quality control value
+! obs - observation type
+!
+! created Oct. 2007 Ryan Torn, NCAR/MMM
+! adapted for more generic use 11 Mar 2010, nancy collins, ncar/image
+!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+subroutine create_3d_obs(lat, lon, vval, vkind, obsv, okind, oerr, day, sec, qc, obs)
+use types_mod, only : r8
+use obs_def_mod, only : obs_def_type, set_obs_def_time, set_obs_def_kind, &
+ set_obs_def_error_variance, set_obs_def_location
+use obs_sequence_mod, only : obs_type, set_obs_values, set_qc, set_obs_def
+use time_manager_mod, only : time_type, set_time
+use location_mod, only : set_location
+
+ integer, intent(in) :: okind, vkind, day, sec
+ real(r8), intent(in) :: lat, lon, vval, obsv, oerr, qc
+ type(obs_type), intent(inout) :: obs
+
+real(r8) :: obs_val(1), qc_val(1)
+type(obs_def_type) :: obs_def
+
+call set_obs_def_location(obs_def, set_location(lon, lat, vval, vkind))
+call set_obs_def_kind(obs_def, okind)
+call set_obs_def_time(obs_def, set_time(sec, day))
+call set_obs_def_error_variance(obs_def, oerr * oerr)
+call set_obs_def(obs, obs_def)
+
+obs_val(1) = obsv
+call set_obs_values(obs, obs_val)
+qc_val(1) = qc
+call set_qc(obs, qc_val)
+
+end subroutine create_3d_obs
+
+
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+!
+! add_obs_to_seq -- adds an observation to a sequence. inserts if first
+! obs, inserts with a prev obs to save searching if that's possible.
+!
+! seq - observation sequence to add obs to
+! obs - observation, already filled in, ready to add
+! obs_time - time of this observation, in dart time_type format
+! prev_obs - the previous observation that was added to this sequence
+! (will be updated by this routine)
+! prev_time - the time of the previously added observation (will also
+! be updated by this routine)
+! first_obs - should be initialized to be .true., and then will be
+! updated by this routine to be .false. after the first obs
+! has been added to this sequence.
+!
+! created Mar 8, 2010 nancy collins, ncar/image
+!
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+subroutine add_obs_to_seq(seq, obs, obs_time, prev_obs, prev_time, first_obs)
+ use types_mod, only : r8
+ use obs_sequence_mod, only : obs_sequence_type, obs_type, insert_obs_in_seq
+ use time_manager_mod, only : time_type, operator(>=)
+
+ type(obs_sequence_type), intent(inout) :: seq
+ type(obs_type), intent(inout) :: obs, prev_obs
+ type(time_type), intent(in) :: obs_time
+ type(time_type), intent(inout) :: prev_time
+ logical, intent(inout) :: first_obs
+
+! insert(seq,obs) always works (i.e. it inserts the obs in
+! proper time format) but it can be slow with a long file.
+! supplying a previous observation that is older (or the same
+! time) as the new one speeds up the searching a lot.
+
+if(first_obs) then ! for the first observation, no prev_obs
+ call insert_obs_in_seq(seq, obs)
+ first_obs = .false.
+else
+ if(obs_time >= prev_time) then ! same time or later than previous obs
+ call insert_obs_in_seq(seq, obs, prev_obs)
+ else ! earlier, search from start of seq
+ call insert_obs_in_seq(seq, obs)
+ endif
+endif
+
+! update for next time
+prev_obs = obs
+prev_time = obs_time
+
+end subroutine add_obs_to_seq
+
+end program text_to_obs
Property changes on: DART/trunk/observations/text/text_to_obs.f90
___________________________________________________________________
Added: svn:mime-type
+ text/plain
Added: svn:keywords
+ Date Rev Author HeadURL Id
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/text_to_obs.html
===================================================================
--- DART/trunk/observations/text/text_to_obs.html (rev 0)
+++ DART/trunk/observations/text/text_to_obs.html 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,153 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
+ "http://www.w3.org/TR/html4/strict.dtd">
+<HTML>
+<HEAD>
+<TITLE>MADIS Data</TITLE>
+<link rel="stylesheet" type="text/css" href="../../doc/html/doc.css">
+</HEAD>
+<BODY>
+<A NAME="TOP"></A>
+
+<center>
+<A HREF="#Overview">OVERVIEW</A> /
+<A HREF="#DataSources">DATA SOURCES</A> /
+<A HREF="#Programs">PROGRAMS</A> /
+<A HREF="#KnownBugs">KNOWN BUGS</A> /
+<A HREF="#FuturePlans">FUTURE PLANS</A> /
+<A HREF="#Legalese">TERMS OF USE</A>
+</center>
+
+<H1>Simple Text File to DART Converter</H1>
+<small><P>
+<!-- version tag follows, do not edit -->$Id$
+</P></small>
+
+<P>
+If you have observations in spreadsheet or column format, in text,
+with a single line per observation, then the files this directory
+are a template for how to convert these observations
+into a format suitable for DART use.
+</P>
+<P>
+The workflow is you collect the needed information about each
+observation - location, time, data value, observation type - from
+your data source, and then
+call a series of DART library routines to construct a derived type that
+contains all the observation information. Then another set of DART library
+routines puts them into a time-sorted series. And finally, the last
+routine is a write subroutine that writes out the series to a file.
+It is not recommended that you try to mimic the ascii file format
+by other means; the format is subject to change and the library routines
+will continue to be supported even if the physical format changes.
+</P>
+<P>
+If your input data is in some kind of format like netCDF or HDF,
+then one of the other converters (the MADIS ones for netCDF) might
+be a better starting place for adapting code.
+</P>
+
+
+<!--==================================================================-->
+
+<A NAME="DataSources"></A>
+<HR>
+<H2>DATA SOURCES</H2>
+
+<P>
+This part is up to you. You will need a location, a value,
+a type, and some kind of error estimate. The error estimate
+can be hardcoded in the converter if it's not available in
+the input data.
+</P>
+
+<!--==================================================================-->
+
+<A NAME="Programs"></A>
+<HR>
+<H2>PROGRAMS</H2>
+<P>
+The <em class=file>text_to_obs.f90</em> file is the main converter.
+Look at the source code where it reads the sample data file. You will
+almost certainly need to change the read to match your data. The
+sample code provided reads in each text line into a character buffer
+and then uses reads from that buffer to parse up the data items.
+</P>
+<P>
+To compile and test,
+go into the work subdirectory and run the <em class=file>quickbuild.csh</em>
+script to build the converter and a couple of general purpose utilities.
+<em class=file>advance_time</em> helps compute times, and
+<em class=file>obs_sequence_tool</em> manipulates DART observation files
+once they have been created.
+</P>
+<P>
+To add different observation types, look in the
+<em class=file>DART/obs_def</em> directory. If you can
+find an obs_def_XXX_mod.f90 file with an appropriate set
+of observations, put that name in the <em class=file>input.nml</em>
+namelist file, in the &preprocess_nml namelist, 'input_files'.
+Multiples can be listed. Then run quickbuild.csh again to remake
+the table of supported observation types before trying to recompile
+the source code.
+</P>
+<P>
+An example script for converting batches of files is
+in the <em class=file>shell_scripts</em> directory. A tiny example
+data file is in the <em class=file>data</em> directory.
+These are <em>NOT</em> intended to be turnkey scripts; they will
+certainly need to be customized for your use. There are comments
+at the top of the script saying what options they include, and
+should be commented enough to indicate where changes will be
+likely to need to be made.
+</P>
+
+<!--==================================================================-->
+<!-- Describe the bugs. -->
+<!--==================================================================-->
+
+<A NAME="KnownBugs"></A>
+<HR>
+<H2>KNOWN BUGS</H2>
+<P>
+none
+</P>
+
+<!--==================================================================-->
+<!-- Describe Future Plans. -->
+<!--==================================================================-->
+
+<A NAME="FuturePlans"></A>
+<HR>
+<H2>FUTURE PLANS</H2>
+<P>
+none
+</P>
+
+<!--==================================================================-->
+<!-- Legalese & Metadata -->
+<!--==================================================================-->
+
+<A NAME="Legalese"></A>
+<HR>
+<H2>Terms of Use</H2>
+
+<P>
+DART software - Copyright © 2004 - 2010 UCAR.<br>
+This open source software is provided by UCAR, "as is",<br>
+without charge, subject to all terms of use at<br>
+<a href="http://www.image.ucar.edu/DAReS/DART/DART_download">
+http://www.image.ucar.edu/DAReS/DART/DART_download</a>
+</P>
+
+<TABLE border=0 cellpadding=0 width=100% summary="">
+<TR><TD valign=top>Contact: </TD><TD> nancy collins </TD></TR>
+<TR><TD valign=top>Revision: </TD><TD> $Revision$ </TD></TR>
+<TR><TD valign=top>Source: </TD><TD> $URL$ </TD></TR>
+<TR><TD valign=top>Change Date: </TD><TD> $Date$ </TD></TR>
+<TR><TD valign=top>Change history: </TD><TD> try "svn log" or "svn diff" </TD></TR>
+</TABLE>
+
+<!--==================================================================-->
+
+</BODY>
+</HTML>
Property changes on: DART/trunk/observations/text/text_to_obs.html
___________________________________________________________________
Added: svn:mime-type
+ text/html
Added: svn:keywords
+ Date Rev Author HeadURL Id
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/work/input.nml
===================================================================
--- DART/trunk/observations/text/work/input.nml (rev 0)
+++ DART/trunk/observations/text/work/input.nml 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,23 @@
+
+&preprocess_nml
+ input_obs_kind_mod_file = '../../../obs_kind/DEFAULT_obs_kind_mod.F90',
+ output_obs_kind_mod_file = '../../../obs_kind/obs_kind_mod.f90',
+ input_obs_def_mod_file = '../../../obs_def/DEFAULT_obs_def_mod.F90',
+ output_obs_def_mod_file = '../../../obs_def/obs_def_mod.f90',
+ input_files = '../../../obs_def/obs_def_eval_mod.f90',
+ /
+
+&obs_kind_nml
+ /
+
+&location_nml
+ /
+
+&utilities_nml
+ module_details = .false.
+ /
+
+&obs_sequence_nml
+ write_binary_obs_sequence = .false.
+ /
+
Property changes on: DART/trunk/observations/text/work/input.nml
___________________________________________________________________
Added: svn:mime-type
+ text/plain
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/work/mkmf_advance_time
===================================================================
--- DART/trunk/observations/text/work/mkmf_advance_time (rev 0)
+++ DART/trunk/observations/text/work/mkmf_advance_time 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,18 @@
+#!/bin/csh
+#
+# DART software - Copyright \xA9 2004 - 2010 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# $Id$
+
+../../../mkmf/mkmf -p advance_time -t ../../../mkmf/mkmf.template -c"-Duse_netCDF" \
+ -a "../../.." path_names_advance_time
+
+exit $status
+
+# <next few lines under version control, do not edit>
+# $URL$
+# $Revision$
+# $Date$
+
Property changes on: DART/trunk/observations/text/work/mkmf_advance_time
___________________________________________________________________
Added: svn:executable
+ *
Added: svn:mime-type
+ text/plain
Added: svn:keywords
+ Date Rev Author HeadURL Id
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/work/mkmf_obs_sequence_tool
===================================================================
--- DART/trunk/observations/text/work/mkmf_obs_sequence_tool (rev 0)
+++ DART/trunk/observations/text/work/mkmf_obs_sequence_tool 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,18 @@
+#!/bin/csh
+#
+# DART software - Copyright \xA9 2004 - 2010 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# $Id$
+
+../../../mkmf/mkmf -p obs_sequence_tool -t ../../../mkmf/mkmf.template -c"-Duse_netCDF" \
+ -a "../../.." path_names_obs_sequence_tool
+
+exit $status
+
+# <next few lines under version control, do not edit>
+# $URL$
+# $Revision$
+# $Date$
+
Property changes on: DART/trunk/observations/text/work/mkmf_obs_sequence_tool
___________________________________________________________________
Added: svn:executable
+ *
Added: svn:mime-type
+ text/plain
Added: svn:keywords
+ Date Rev Author HeadURL Id
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/work/mkmf_preprocess
===================================================================
--- DART/trunk/observations/text/work/mkmf_preprocess (rev 0)
+++ DART/trunk/observations/text/work/mkmf_preprocess 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,18 @@
+#!/bin/csh
+#
+# DART software - Copyright \xA9 2004 - 2010 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# $Id$
+
+../../../mkmf/mkmf -p preprocess -t ../../../mkmf/mkmf.template -c"-Duse_netCDF" \
+ -a "../../.." path_names_preprocess
+
+exit $status
+
+# <next few lines under version control, do not edit>
+# $URL$
+# $Revision$
+# $Date$
+
Property changes on: DART/trunk/observations/text/work/mkmf_preprocess
___________________________________________________________________
Added: svn:executable
+ *
Added: svn:mime-type
+ text/plain
Added: svn:keywords
+ Date Rev Author HeadURL Id
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/work/mkmf_text_to_obs
===================================================================
--- DART/trunk/observations/text/work/mkmf_text_to_obs (rev 0)
+++ DART/trunk/observations/text/work/mkmf_text_to_obs 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,18 @@
+#!/bin/csh
+#
+# DART software - Copyright \xA9 2004 - 2010 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# $Id$
+
+../../../mkmf/mkmf -p text_to_obs -t ../../../mkmf/mkmf.template -c"-Duse_netCDF" \
+ -x -a "../../.." path_names_text_to_obs
+
+exit $status
+
+# <next few lines under version control, do not edit>
+# $URL$
+# $Revision$
+# $Date$
+
Property changes on: DART/trunk/observations/text/work/mkmf_text_to_obs
___________________________________________________________________
Added: svn:executable
+ *
Added: svn:mime-type
+ text/plain
Added: svn:keywords
+ Date Rev Author HeadURL Id
Added: svn:eol-style
+ native
Added: DART/trunk/observations/text/work/path_names_advance_time
===================================================================
--- DART/trunk/observations/text/work/path_names_advance_time (rev 0)
+++ DART/trunk/observations/text/work/path_names_advance_time 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,6 @@
+time_manager/advance_time.f90
+time_manager/time_manager_mod.f90
+common/types_mod.f90
+utilities/utilities_mod.f90
+utilities/parse_args_mod.f90
+mpi_utilities/null_mpi_utilities_mod.f90
Added: DART/trunk/observations/text/work/path_names_obs_sequence_tool
===================================================================
--- DART/trunk/observations/text/work/path_names_obs_sequence_tool (rev 0)
+++ DART/trunk/observations/text/work/path_names_obs_sequence_tool 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,14 @@
+obs_sequence/obs_sequence_tool.f90
+obs_sequence/obs_sequence_mod.f90
+obs_kind/obs_kind_mod.f90
+obs_def/obs_def_mod.f90
+cov_cutoff/cov_cutoff_mod.f90
+assim_model/assim_model_mod.f90
+models/template/model_mod.f90
+common/types_mod.f90
+location/threed_sphere/location_mod.f90
+mpi_utilities/null_mpi_utilities_mod.f90
+random_seq/random_seq_mod.f90
+random_nr/random_nr_mod.f90
+time_manager/time_manager_mod.f90
+utilities/utilities_mod.f90
Added: DART/trunk/observations/text/work/path_names_preprocess
===================================================================
--- DART/trunk/observations/text/work/path_names_preprocess (rev 0)
+++ DART/trunk/observations/text/work/path_names_preprocess 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,5 @@
+preprocess/preprocess.f90
+common/types_mod.f90
+utilities/utilities_mod.f90
+mpi_utilities/null_mpi_utilities_mod.f90
+time_manager/time_manager_mod.f90
Added: DART/trunk/observations/text/work/path_names_text_to_obs
===================================================================
--- DART/trunk/observations/text/work/path_names_text_to_obs (rev 0)
+++ DART/trunk/observations/text/work/path_names_text_to_obs 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,13 @@
+observations/text/text_to_obs.f90
+location/threed_sphere/location_mod.f90
+obs_sequence/obs_sequence_mod.f90
+obs_kind/obs_kind_mod.f90
+obs_def/obs_def_mod.f90
+assim_model/assim_model_mod.f90
+models/template/model_mod.f90
+common/types_mod.f90
+random_seq/random_seq_mod.f90
+random_nr/random_nr_mod.f90
+utilities/utilities_mod.f90
+time_manager/time_manager_mod.f90
+mpi_utilities/null_mpi_utilities_mod.f90
Added: DART/trunk/observations/text/work/quickbuild.csh
===================================================================
--- DART/trunk/observations/text/work/quickbuild.csh (rev 0)
+++ DART/trunk/observations/text/work/quickbuild.csh 2010-03-30 15:11:11 UTC (rev 4331)
@@ -0,0 +1,69 @@
+#!/bin/csh
+#
+# DART software - Copyright \xA9 2004 - 2010 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# $Id$
+#
+# compile all converter programs
+
+#----------------------------------------------------------------------
+# 'preprocess' is a program that culls the appropriate sections of the
+# observation module for the observations types in 'input.nml'; the
+# resulting source file is used by all the remaining programs,
+# so this MUST be run first.
+#----------------------------------------------------------------------
+
+\rm -f preprocess *.o *.mod
+\rm -f ../../../obs_def/obs_def_mod.f90
+\rm -f ../../../obs_kind/obs_kind_mod.f90
+
+set MODEL = "text converters"
+
+@ n = 1
+
+echo
+echo
+echo "---------------------------------------------------------------"
+echo "${MODEL} build number ${n} is preprocess"
+
+csh mkmf_preprocess
+make || exit $n
+
+./preprocess || exit 99
+
+#----------------------------------------------------------------------
+# Build all the single-threaded targets
+#----------------------------------------------------------------------
+
+foreach TARGET ( mkmf_* )
+
+ set PROG = `echo $TARGET | sed -e 's#mkmf_##'`
+
+ switch ( $TARGET )
+ case mkmf_preprocess:
+ breaksw
+ default:
+ @ n = $n + 1
+ echo
+ echo "---------------------------------------------------"
+ echo "${MODEL} build number ${n} is ${PROG}"
+ \rm -f ${PROG}
+ csh $TARGET || exit $n
+ make || exit $n
+ breaksw
+ endsw
+end
+
+\rm -f *.o *.mod input.nml*_default
+
+echo "Success: All ${MODEL} programs compiled."
+
+exit 0
+
+# <next few lines under version control, do not edit>
+# $URL$
+# $Revision$
+# $Date$
+
Property changes on: DART/trunk/observations/text/work/quickbuild.csh
___________________________________________________________________
Added: svn:executable
+ *
Added: svn:mime-type
@@ Diff output truncated at 40000 characters. @@
More information about the Dart-dev
mailing list