+If you are going to run CESM/DART on the NCAR "yellowstone" machine, +here are some canned instructions to get you started: +

Get CESM

+Check out CESM1.1.1 which is one release back +from the very most current version (they updated versions +right before the Breckenridge User's meeting). +We will soon have a chance to test the code with +the latest release, but for now we know this setup +works with 1.1.1. +

+You will want "cesm1_1_1" as the model version. You will have +to register as a user to check out the source code. There is a +default username and password to access the subversion website. +

+This web page shows the names of the 'compsets' which are the +configurations of the various models:
+ +

+If you want to try to build CESM separately from DART +to learn about it, that's fine. But our scripts will need +to rebuild it again with the DART configuration so you can +wait on building it if you want. +

Required patches to CESM1.1.1

+There are a few CESM files that we still need to modify to make +things work with DART. In this dir on yellowstone is a tar file:

DART

+Go to the DART web pages: +http://www.image.ucar.edu/DARES/ +and register as a DART user. It's important for us to be able +to show as large a user base as possible and everyone who +registers helps us out. +

+Check out the "trunk" version. The following instructions +assume you've checked it out into a subdirectory "DART". +

+In the DART/mkmf directory copy the +mkmf.template.intel.linux to +mkmf.template . That should work +for yellowstone with the default set of modules that are loaded +on that machine. **IMPORTANT** if you +are ever thinking of running with an active ocean, you must add:
+

+If you want to be able to reproduce runs exactly, you need to make +sure this flag is part of the FFLAGS line:
+

+There are some things you can customize at the top, including the compset, +your project number for running, how many ensemble members, etc. make +any changes you want there. +

+Run the script. It will build the CAM, POP, and CLM filters first, and then +build and configure CESM. When the script is done you should have +files in several directories:
+

EVERYTHING BELOW HERE PERTAINS ONLY TO POP AND NEEDS UPDATING

CESM1.0 POP2

From nancy at ucar.edu Thu Aug 1 10:20:07 2013 From: nancy at ucar.edu (nancy at ucar.edu) Date: Thu, 01 Aug 2013 10:20:07 -0600 Subject: [Dart-dev] [6354] DART/trunk/doc/html: New docs for the upcoming release. Message-ID: Revision: 6354 Author: nancy Date: 2013-08-01 10:20:07 -0600 (Thu, 01 Aug 2013) Log Message: ----------- New docs for the upcoming release. Reformatted information from the wiki pages into html for the release differences file. The release notes file has only minor updates so far, e.g. changing names from Jamaica/Kodiak to Kodiak/Lanai. It needs more work but i want a version in the repository first. Added Paths: ----------- DART/trunk/doc/html/Lanai_diffs_from_Kodiak.html DART/trunk/doc/html/Lanai_release.html -------------- next part -------------- Added: DART/trunk/doc/html/Lanai_diffs_from_Kodiak.html =================================================================== --- DART/trunk/doc/html/Lanai_diffs_from_Kodiak.html (rev 0) +++ DART/trunk/doc/html/Lanai_diffs_from_Kodiak.html 2013-08-01 16:20:07 UTC (rev 6354) @@ -0,0 +1,297 @@ + + + +DART Lanai Differences from Kodiak Release Notes + + + + + + +

DART Lanai Differences from Kodiak Release Notes

+ + + + + + +

+ +	+ Jump to DART Documentation Main Index + Website + or local file + version information for this file: + + $Id$ +

+ +New Models or Changes to Existing Models / +Changes to Core DART routines / +New or Changed Forward Operators / +Observation Converters / +New or Updated DART Diagnostics / +Tutorial, Scripting, Setup, Builds / +Terms of Use + + + +

[top]

+ +

New Models or Changes to Existing Models

+Several new models have been incorporated into DART. This section details +both changes to existing models and descriptions of new models that have +been added since the Kodiak release. +

+ +

Changes to the WRF model_mod: +
- allow advanced microphysics schemes (needed interpolation for 7 new kinds)
- option to interpolate in the vertical in log(p) instead of linear in p (log(p) is the default now)
- added support in the namelist to avoid writing updated fields back into the wrf netcdf files. The fields are still updated during the assimilation but the updated data is not written back to the wrfinput file during the dart_to_wrf step.
- fixed an obscure bug in the vertical convert routine of the wrf model_mod that would occasionally fail to convert an obs. This would make tiny differences in the output as the number of mpi tasks change. No quantitative differences in the output but not bitwise compatible before and is again now.
+
Added support for the MPAS_ATM and MPAS_OCN models. +
- added code to the mpas_atm model to interpolate specific humidity and pressure, so we can assimilate GPS obs now.
+
Added support for the CLM land model.
Added support for the CESM 1.1.1 release for CAM, POP, CLM: includes experiment setup scripts and assimilation scripts.
Added support for the 'SQG' uniform PV two-surface QC+1 spectral model.
Added support for a flux-transport solar dynamo model.
Added support for the NOAH land model.
Added support for the NAAPS model.
Added model_mod interface code for the NOGAPS model to the SVN repository.
Fix temperature vs potential temp bug in POP model_mod. +
- state vector has all along contained potential temperature and not in-situ (sensible) temperature. The observations from the World Ocean Database are of sensible temperature. Changed the specific kind in the model_mod to be
```
KIND_POTENTIAL_TEMPERATURE
```
  and added new code to convert from potential to in-situ temperature. Differences for even the deeper obs (4-5km) is still small ( ~ 0.2 degree). (in-situ or sensible temperature is what you measure with a regular thermometer.)
+
Simple advection model: +
- Fix where the random number seed is set in the models/simple_advection model_mod - it needed to be sooner than it was being called.
+

+ + + +

[top]

+ +

Changes to Core DART routines

+This section describes changes in the basic DART library routines +since the Kodiak release. +

+ +

Added a completely new random number generator based on the Mersenne Twister algorithm from the GNU scientific library. It seems to have better behavior if reseeded frequently, which is a possible usage pattern if perfect_model_obs is run for only single steps and the model is advanced in an external script. As part of this code update all random number code was moved into the random_seq_mod and random_nr_mod is deprecated.
perfect_model_obs calls a seed routine in the time manager now that generates a consistent seed based on the current time of the state. This makes subsequent runs give consistent results and yet separate runs don't get identical error values.
Added random number generator seeds in several routines to try to get consistent results no matter how many MPI tasks the code was run with. This includes: +
- cam model_mod.f90, pert_model_state()
- assim_tools_mod.f90, filter_assim(), filter kinds 2, 3, and 5
- wrf model_mod.f90, pert_model_state()
- adaptive_inflate_mod.f90, adaptive_inflate_init(), non-deterministic inf
+
Added a time sort routine in the time_manager_mod.
Avoid a pair of all-to-all transposes when setting the inflation mean and sd from the namelist. The new code finds the task which has the two copies and sets them directly without a transpose. The log messages were also moved to the end of the routine - if you read in the mean/sd values from a restart file the log messages that printed out the min/max values needed to be after the read from the file.
Reordered the send/receive loops in the all-to-all transposes to scale better on yellowstone.
Remove a state-vector size array from the stack in read_ensemble_restart(). The array is allocated only if needed and then deallocated. The ensemble write routine was changed before the Kodiak release but the same code in read was apparently not changed simply as an oversight.
If the ensemble mean is selected to be written out in dart restart file format, the date might not have been updated correctly. The code was fixed to ensure the ensemble mean date in the file was correct.
filter writes the ensemble size into the log file.
Reorganized the code in the section of obs_model_mod that prints out the time windows, with and without verbose details. Should be clearer if the next observation is in or out of the current assimilation window, and if the model needs to advance or not.
Added a fill_inflation_restart utility which can write a file with a fixed mean and sd, so the first step of a long assimilation run can use the same 'start_from_restart_file' as subsequent steps.
Added new location module options: +
- Channel coordinate system
- [0-1] periodic 3D coordinate system
- X,Y,Z 3D Cartesian coordinate system
- 2D annulus coordinate system
+

+ + + + +

[top]

+ +

New or changed Forward Operators

+This section describes changes to the Foward Operators and +new Generic Kinds or Specific Types that have been added +since the Kodiak release. +

+ + +

Many new kinds added to the DEFAULT_obs_kind_mod.f90: +
- KIND_CANOPY_WATER
- KIND_CARBON
- KIND_CLW_PATH
- KIND_DIFFERENTIAL_REFLECTIVITY
- KIND_DUST
- KIND_EDGE_NORMAL_SPEED
- KIND_FLASH_RATE_2D
- KIND_GRAUPEL_VOLUME
- KIND_GROUND_HEAT_FLUX KIND_HAIL_MIXING_RATIO
- KIND_HAIL_NUMBER_CONCENTR
- KIND_HAIL_VOLUME KIND_ICE KIND_INTEGRATED_AOD
- KIND_INTEGRATED_DUST
- KIND_INTEGRATED_SEASALT KIND_INTEGRATED_SMOKE
- KIND_INTEGRATED_SULFATE
- KIND_LATENT_HEAT_FLUX
- KIND_LEAF_AREA_INDEX
- KIND_LEAF_CARBON
- KIND_LEAF_NITROGEN KIND_LIQUID_WATER
- KIND_MICROWAVE_BRIGHT_TEMP
- KIND_NET_CARBON_FLUX
- KIND_NET_CARBON_PRODUCTION
- KIND_NEUTRON_INTENSITY
- KIND_NITROGEN KIND_RADIATION
- KIND_ROOT_CARBON
- KIND_ROOT_NITROGEN
- KIND_SEASALT
- KIND_SENSIBLE_HEAT_FLUX
- KIND_SMOKE
- KIND_SNOWCOVER_FRAC
- KIND_SNOW_THICKNESS
- KIND_SNOW_WATER
- KIND_SO2
- KIND_SOIL_CARBON
- KIND_SOIL_NITROGEN
- KIND_SPECIFIC_DIFFERENTIAL_PHASE
- KIND_STEM_CARBON
- KIND_STEM_NITROGEN
- KIND_SULFATE
- KIND_VORTEX_WMAX
- KIND_WATER_TABLE_DEPTH
- KIND_WIND_TURBINE_POWER
- plus slots 151-250 reserved for Chemistry (specifically WRF-Chem) kinds
+
Added a forward operator for total precipitable water. It loops over model levels so it can be used as an example of how to handle this without having to hardcode the number of levels into the operator.
Added a forward operator (and obs_seq file converter) for COSMOS ground moisture observations.
Added a forward operator (and obs_seq file converter) for MIDAS observations of Total Electron Count.
Added a 'set_1d_integral()' routine to the obs_def_1d_state_mod.f90 forward operator for the low order models. This subroutine isn't used by filter but it would be needed if someone wanted to write a standalone program to generate obs of this type. We use this file as an example of how to write an obs type that has metadata, but we need to give an example of how to set the metadata if you aren't using create_obs_sequence interactively (e.g. your data is in netcdf and you have a separate converter program.)

+ + + + +

[top]

+ +

Observation Converters

+This section describes +support for new observation types or sources +that have been added since the Kodiak release. +

+ + +

Added an obs_sequence converter for wind profiler data from MADIS.
Added an obs_sequence converter for Ameriflux land observations(latent heat flux, sensible heat flux, net ecosystem production).
Added an obs_sequence converter for MODIS snow coverage measurements.
Added an obs_sequence converter for COSMOS ground moisture observations.
Added an obs_sequence converter for MIDAS observations of Total Electron Count.
Updated scripts for the GPS converter; added options to convert data from multiple satellites.
More scripting support in the MADIS obs converters; more error checks added to the rawin converter.
Added processing for wind profiler observation to the wrf_dart_obs_preprocess program.
Fix BUG in airs converter - the humidity obs are accumulated acrossthe layers and so the best location for them is the layer midpoint and not on the edges (levels) as the temperature obs are. Also fixed off-by-one error where the converter would make one more obs above the requested top level.
Made gts_to_dart converter create separate obs types for surface dewpoint vs obs aloft because they have different vertical coordinates.
Converted mss commands to hpss commands for a couple observation converter shell scripts (inc AIRS).
New matlab code to generate evenly spaced observations on the surface of a sphere (e.g. the globe).
Added obs_loop.f90 example file in obs_sequence directory; example template for how to construct special purpose obs_sequence tools.
Change the default in the script for the prepbufr converter so it will swap bytes, since all machines except ibms will need this now.

+ + + + +

[top]

+ +

New or updated DART Diagnostics

+This section describes new or updated diagnostic +routines that have been added since the Kodiak release. +

+ +

Handle empty epochs in the obs_seq_to_netcdf converter.
Added a matlab utility to show the output of a 'hop' test (running a model for a continuous period vs. stopping and restarting a run).
The obs_common_subset program can select common observations from up to 4 observation sequence files at a time.
Add code in obs_seq_verify to ensure that the ensemble members are in the same order in all netcdf files.
Added support for the unstructured grids of mpas to our matlab diagnostics.
Fix to writing of ReportTime in obs_seq_coverage.
Fixed logic in obs_seq_verify when determining the forecast lat.
Fixed loops inside obs_seq_coverage which were using the wrong limits on the loops. Fixed writing of 'ntimes' in output netcdf variable.
Rewrote the algorithm in the obs_selection tool so it had better scaling with large numbers of obs.
Added preliminary support for a list of 'trusted obs' in the obs_diag program. Can disable the rank histogram generation with a namelist item. can define height_edges or heights in the namelist, but not both.
Extend obs_seq_verify so it can be used for forecasts from a single member. minor changes to obs_selection, obs_seq_coverage and obs_seq_verify to support a single member.
Added Matlab script to read/print timestamps from binary dart restart/ic files.
Default for obs_seq_to_netcdf in all the namelists is now 'one big time bin' so you don't have to know the exact timespan of an obs_seq.final file before converting to netCDF.

+ + + + +

[top]

+ +

Tutorial, Scripting, Setup, Builds

+This section describes +updates and changes to the tutorial materials, +scripting, setup, and build information +since the Kodiak release. +

+ + +

Make the default input.nml for the Lorenz 96 and Lorenz 63 model gives good assimilation results. Rename the original input.nml to input.workshop.nml. The workshop_setup script renames it back before doing anything else so this won't break the workshop instructions. Simplify all the workshop_setup.csh scripts to do the minimal work needed by the DART tutorial.
Updates to the models/template directory with the start of a full 3d geophysical model template. Still under construction.
Move the pdf files in the tutorial directory up a level. routines that generate diagrams/figures are moved to the DAReS/group/DART_tutorial dir. framemaker files removed because we don't have a working version to update or read them with.
Enable netCDF large file support in the work/input.nml for models which are likely to have large state vectors.
Minor updates to the doc.css file, make pages look identical in the safari and firefox browsers.
Added a utility that sorts and reformats namelists, culls all comments to the bottom of the file. Useful for doing diffs and finding duplicated namelists in a file.
Cleaned up mkmf files - removed files for obsolete platforms and compilers, updated suggested default flags for intel.
Update the mkmf template for gfortran to allow fortran source lines longer than 132 characters.

+ + + + + + +

[top]

+ +

Terms of Use

+ +

+DART software - Copyright 2004 - 2013 UCAR.
+This open source software is provided by UCAR, "as is",
+without charge, subject to all terms of use at
+ +http://www.image.ucar.edu/DAReS/DART/DART_download +

+ + + + + + + +

Contact:	DART core group
Revision:	$Revision$
Source:	$URL$
Change Date:	$Date$
Change history:	try "svn log" or "svn diff"

+ + + + + Property changes on: DART/trunk/doc/html/Lanai_diffs_from_Kodiak.html ___________________________________________________________________ Added: svn:mime-type + text/html Added: svn:keywords + Date Rev Author HeadURL Id Added: svn:eol-style + native Added: DART/trunk/doc/html/Lanai_release.html =================================================================== --- DART/trunk/doc/html/Lanai_release.html (rev 0) +++ DART/trunk/doc/html/Lanai_release.html 2013-08-01 16:20:07 UTC (rev 6354) @@ -0,0 +1,3189 @@ + + + +DART Lanai Release Notes + + + + + + +

DART Lanai Release Notes

+ + + + + + +

+ +	+ Jump to DART Documentation Main Index + Website + or local file + version information for this file: + + $Id$ +

+ +Dart Overview / +Getting Started / +Installation / +Notes for Current Users / +Non-backwards Compatible Changes / +New Features / +New Models / +Changed Models / +New Observations / +New Diagnostics and Documentation / +New Utilities / +Known Problems / +Terms of Use + + + + +

Dart Overview

+ +

The Data Assimilation Research Testbed (DART) is designed to +facilitate the combination of assimilation algorithms, models, +and real (or synthetic) observations to allow +increased understanding of all three. +The DART programs are highly portable, having been +compiled with many Fortran 90 compilers +and run on linux compute-servers, linux clusters, OSX laptops/desktops, +SGI Altix clusters, supercomputers running AIX, and more. +Read the +Customizations section +for help in building on new platforms.

+ +

+DART employs a modular programming approach to apply an Ensemble Kalman Filter +which adjusts model values toward a state that is more consistent with information +from a set of observations. Models may be swapped in and out, as can +different algorithms in the Ensemble Kalman Filter. The method +requires running multiple instances of a model to generate an ensemble of +states. A forward operator appropriate for the type of observation being assimilated +is applied to each of the states to generate the model's estimate of the observation. +Comparing these estimates and their uncertainty to the observation and +its uncertainty ultimately results in the adjustments to the model states. +See the DARTLAB demos or read more in the tutorials included with the +DART distribution. They are described below.

+ +

+DART diagnostic output includes two netCDF files containing +the model states just before +the adjustment (Prior_Diag.nc) and just after the adjustment +(Posterior_Diag.nc) as well as a file +obs_seq.final with the model estimates of the observations. +There is a suite of Matlab® functions that facilitate exploration of the +results, but the netCDF files are inherently portable and contain all the +necessary metadata to interpret the contents with other analysis programs +such as NCL, R, etc. +

+ +

In this document links are available which point to Web-based documentation +files and also to the same information in html files distributed with DART. +If you have used subversion to check out a local copy of the DART files you +can open this file in a browser by loading +DART/doc/html/Lanai_release.html +and then use the local file links to see +other documentation pages without requiring a connection to +the internet. +If you are looking at this documentation from +the www.image.ucar.edu web server or you are +connected to the internet you can use the +Website links to view other documentation pages. +

+ + + + +

Getting Started

+ +

What's Required

a Fortran 90 compiler
a netCDF library including the F90 interfaces
the C shell
(optional, to run in parallel) an MPI library

+DART has been tested on many Fortran compilers and platforms. +We don't have any platform-dependent code sections and we use +only the parts of the language that are portable across all +the compilers we have access to. +We explicitly set the Fortran 'kind' for all real values and do +not rely on autopromotion or other compile-time flags to set the +default byte size for numbers. +It is possible that some model-specific interface code from +outside sources may have specific compiler flag requirements; +see the documentation for each model. +The low-order models and all common portions of the DART code +compile cleanly. +
+
+DART uses the +netCDF +self-describing data format with a particular metadata convention to +describe output that is used to analyze the results of assimilation +experiments. These files have the extension .nc +and can be read by a number of standard data analysis tools. +
+
+Since most of the models being used with DART are +written in Fortran and run on various UNIX or *nix platforms, the +development environment for DART is highly skewed to these machines. +We do most of our development on a small linux workstation and a mac laptop +running OSX 10.x, and we have an extensive test network. +(I've never built nor run DART on a Windows machine - so I don't even +know if it's possible. If you have run it (under Cygwin?) please let me +know how it went -- I'm curious. Tim - thoar 'at' ucar 'dot ' edu) +

+ +

What's nice to have

+ +ncview: DART users have used +ncview +to create graphical displays of output data fields. The 2D rendering is +good for 'quick-look' type uses, but I wouldn't want to publish with it. +

+ +NCO: The NCO tools +are able to perform operations on netCDF files like concatenating, slicing, +and dicing.

+ +Matlab®: A set of +Matlab® scripts designed to +produce graphical diagnostics from DART netCDF output files are also part +of the DART project.

+ +MPI: The DART system includes an MPI +option. MPI stands for 'Message Passing Interface', and is both a library and +run-time system that enables multiple copies of a single program to run in +parallel, exchange data, and combine to solve a problem more quickly. +DART does NOT require MPI to run; the default build +scripts do not need nor use MPI in any way. However, for larger models with +large state vectors and large numbers of observations, the data assimilation +step will run much faster in parallel, which requires MPI to be installed and +used. However, if multiple ensembles of your model fit comfortably (in time +and memory space) on a single processor, you need read no further about MPI. +

+ +

Types of input

+ +

DART programs can require three different types of input. +First, some of the DART programs, like those for creating synthetic +observational datasets, require interactive input from the keyboard. +For simple cases this interactive input can be made directly +from the keyboard. In more complicated cases a file containing +the appropriate keyboard input can be created and this file +can be directed to the standard input of the DART program. +Second, many DART programs expect one or more input files in +DART specific formats to be available. For instance, +perfect_model_obs, which creates a synthetic +observation set given a particular model and a description +of a sequence of observations, requires an input file that +describes this observation sequence. +At present, the observation files for DART are in a custom format in either +human-readable ascii or more compact machine-specific binary. +Third, many DART modules (including main programs) make use of +the Fortran90 namelist facility to obtain values of certain parameters +at run-time. All programs look for a namelist input file +called input.nml in the directory in which +the program is executed. The input.nml +file can contain a sequence of individual Fortran90 namelists +which specify values of particular parameters for modules that +compose the executable program. +

+ + + +

Document conventions

+Anything underlined is a URL. +
+
+All filenames look like this -- (typewriter font, green).
+Program names look like this -- (italicized font, green).
+user input looks like this -- (bold, magenta). +

+commands to be typed at the command line are contained in an +indented gray box. +

+And the contents of a file are enclosed in a box with a border: +

+&hypothetical_nml
+  obs_seq_in_file_name = "obs_seq.in",
+  obs_seq_out_file_name = "obs_seq.out",
+  init_time_days = 0,
+  init_time_seconds = 0,
+  output_interval = 1
+&end

+ + + +

[top]

+ +

Installation

+ +

+This document outlines the installation of the DART software +and the system requirements. The entire installation process is summarized in +the following steps: +

+ +

Determine which F90 compiler is available.
Determine the location of the + netCDF library.
Download the DART software + into the expected source tree.
Modify certain DART files to reflect + the available F90 compiler and location of the + appropriate libraries.
Build the executables.

+ +

+We have tried to make the code as portable as possible, but we +do not have access to all compilers on all platforms, so there are no +guarantees. We are interested in your experience building the system, +so please email me (Tim Hoar) thoar 'at' ucar 'dot' edu +(trying to cut down on the spam). +

+ +

+After the installation, you might want to peruse the following. +

+ +

Running the Lorenz_63 Model.
Using the Matlab® diagnostic scripts.
A short discussion on + bias, filter divergence and covariance inflation.
And another one on + synthetic observations.

+ +

You should absolutely run the DARTLAB +interactive tutorial (if you have Matlab available) and look at the +DARTLAB presentation slides + +Website or local file +in the +DART_LAB directory, and then take the tutorial +in the DART/tutorial directory.

+ + + +

[top]

+ +

Requirements: an F90 Compiler

+ +

+The DART software has been successfully built on several Linux/x86 +platforms with several versions of the +Intel Fortran +Compiler for Linux, which (at one point) is/was free for individual +scientific use. Also Intel Fortran for Mac OSX. +It has also been built and successfully run with several +versions of each of the following: +Portland Group Fortran Compiler, +Lahey Fortran Compiler, +Pathscale Fortran Compiler, +GNU Fortran 95 Compiler ("gfortran"), +Absoft Fortran 90/95 Compiler (Mac OSX). +Since recompiling the code is a necessity to experiment +with different models, there are no binaries to distribute. +

+ + + +

[top]

+ +

Requirements: the netCDF library

+ +

+DART uses the +netCDF +self-describing data format for the results of assimilation +experiments. These files have the extension .nc +and can be read by a number of standard data analysis tools. +In particular, DART also makes use of the F90 interface to the library +which is available through the netcdf.mod and +typesizes.mod modules. +IMPORTANT: different compilers create these modules with +different "case" filenames, and sometimes they are not both +installed into the expected directory. It is required that both modules +be present. The normal place would be in the netcdf/include +directory, as opposed to the netcdf/lib directory. +

+ +

+If the netCDF library does not exist on your system, you must build +it (as well as the F90 interface modules). The library and instructions +for building the library or installing from an RPM may be found at +the netCDF home page: + +http://www.unidata.ucar.edu/packages/netcdf/ +Pay particular attention to the compiler-specific patches that must +be applied for the Intel Fortran Compiler. (Or the PG compiler, for +that matter.) +

+ +

+The location of the netCDF library, libnetcdf.a, +and the locations of both netcdf.mod and +typesizes.mod will be needed by the makefile +template, as described in the compiling +section. Depending on the netCDF build options, the Fortran 90 +interfaces may be built in a separate library named +netcdff.a and you may need to add +-lnetcdff to the library flags. +

+ + + +

[top]

+ +

Downloading the distribution.

+ +

+HURRAY! The DART source code is now distributed through +an anonymous Subversion server! The big advantage is +the ability to patch or update existing code trees at your discretion. +Subversion (the client-side app +is 'svn') allows you to compare your code tree with +one on a remote server and selectively update individual files or groups of +files. Furthermore, now everyone has access to any version of any file in +the project, which is a huge help for developers. I have a brief summary of +the svn commands I use most posted at: + +http://www.image.ucar.edu/~thoar/svn_primer.html +

+The resources to develop and support DART come from our ability to +demonstrate our growing user base. We ask that you register at our +download site +http://www.image.ucar.edu/DAReS/DART/DART_download +and promise that the information will only be used to notify you +of new DART releases and shown to our sponsers in an aggregated form: +"Look - we have three users from Tonawanda, NY". After filling in the form, +you will be directed to a website that has instructions on how to download +the code. +

+svn has adopted the strategy that "disk is cheap". In addition to downloading +the code, it downloads an additional copy of the code to store locally (in +hidden .svn directories) as well as some administration files. This allows +svn to perform some commands even when the repository is not available. +It does double the size of the code tree for the initial download, but then +future updates download just the changes, so they usually happen very quickly. +

+If you follow the instructions on the download site, you should wind up with +a directory named DART. Compiling the code in this tree +(as is usually the case) will necessitate much more space. +

+The code tree is very "bushy"; there are many directories of support +routines, etc. but only a few directories involved with the +customization and installation of the DART software. If you can +compile and run ONE of the low-order models, you should be able to +compile and run ANY of the low-order models. For this reason, +we can focus on the Lorenz `63 model. Subsequently, the only +directories with files to be modified to check the installation +are: + DART/mkmf, + DART/models/lorenz_63/work, and + DART/matlab (but only for analysis). +

+ + + +

[top]

+ +

Customizing the build scripts -- Overview.

+ +

+DART executable programs are constructed using two tools: +make and +mkmf. +The make utility is a very common +piece of software that requires a user-defined input file that records +dependencies between different source files. make +then performs a hierarchy of actions when one or more of the +source files is modified. The mkmf utility is +a custom preprocessor that generates a make input file +(named Makefile) and an example namelist +input.nml.program_default +with the default values. The Makefile is designed +specifically to work with object-oriented Fortran90 (and other languages) +for systems like DART. +

+ +

+mkmf requires two separate input files. +The first is a `template' file which specifies details of the commands +required for a specific Fortran90 compiler and may also contain +pointers to directories containing pre-compiled utilities required by +the DART system. This template file will need to +be modified to reflect your system. The second input file is a +`path_names' file which includes a complete list of the locations +(either relative or absolute) of all Fortran90 source files that are +required to produce a particular DART program. +Each 'path_names' file must contain a path for +exactly one Fortran90 file containing a main program, +but may contain any number of additional paths pointing to files +containing Fortran90 modules. +An mkmf command is executed which +uses the 'path_names' file and the mkmf template file to produce a +Makefile which is subsequently used by the +standard make utility. +

+ +

+Shell scripts that execute the mkmf command for all standard +DART executables are provided as part of the standard DART software. +For more information on mkmf see + +the FMS mkmf description. +
+One of the benefits of using mkmf is that it also +creates an example namelist file for each program. The example namelist is +called +input.nml.program_default, +so as not to clash with any +exising input.nml that may exist in that directory. +

+ + +

Building and Customizing the 'mkmf.template' file

+ +

+A series of templates for different compilers/architectures exists +in the DART/mkmf/ directory and have names with +extensions that identify the compiler, the architecture, or both. +This is how you inform the build process of the specifics of your system. +Our intent is that you copy one that is similar to your system into +mkmf.template and customize it. +For the discussion that follows, knowledge of the contents of one of these +templates (i.e. mkmf.template.gfortran) is needed. +Note that only the LAST lines are shown here, +the head of the file is just a big comment (worth reading, btw). +

+ +

+...
+MPIFC = mpif90
+MPILD = mpif90
+FC = gfortran
+LD = gfortran
+NETCDF = /usr/local
+INCS = ${NETCDF}/include
+FFLAGS = -O2 -I$(INCS)
+LIBS = -L${NETCDF}/lib -lnetcdf
+LDFLAGS = -I$(INCS) $(LIBS)
+

+ +

+Essentially, each of the lines defines some part of the resulting +Makefile. Since make +is particularly good at sorting out dependencies, the order of these +lines really doesn't make any difference. +The FC = gfortran line ultimately defines the +Fortran90 compiler to use, etc. +The lines which are most likely to need site-specific changes +start with FFLAGS and NETCDF, which @@ Diff output truncated at 40000 characters. @@ From nancy at ucar.edu Thu Aug 1 13:37:08 2013 From: nancy at ucar.edu (nancy at ucar.edu) Date: Thu, 01 Aug 2013 13:37:08 -0600 Subject: [Dart-dev] [6362] DART/trunk/observations/text/work/mkmf_text_to_obs: this one was accidently committed with the -x flag Message-ID: Revision: 6362 Author: nancy Date: 2013-08-01 13:37:08 -0600 (Thu, 01 Aug 2013) Log Message: ----------- this one was accidently committed with the -x flag (which runs make directly from the mkmf command). it might be a good idea but it's inconsistent with all the other mkmf_xxx files so remove it for now. Modified Paths: -------------- DART/trunk/observations/text/work/mkmf_text_to_obs -------------- next part -------------- Modified: DART/trunk/observations/text/work/mkmf_text_to_obs =================================================================== --- DART/trunk/observations/text/work/mkmf_text_to_obs 2013-08-01 17:44:58 UTC (rev 6361) +++ DART/trunk/observations/text/work/mkmf_text_to_obs 2013-08-01 19:37:08 UTC (rev 6362) @@ -7,7 +7,7 @@ # DART $Id$ ../../../mkmf/mkmf -p text_to_obs -t ../../../mkmf/mkmf.template \ - -x -a "../../.." path_names_text_to_obs + -a "../../.." path_names_text_to_obs exit $status From nancy at ucar.edu Thu Aug 1 13:48:45 2013 From: nancy at ucar.edu (nancy at ucar.edu) Date: Thu, 01 Aug 2013 13:48:45 -0600 Subject: [Dart-dev] [6363] DART/trunk/observations: found a few other mkmf_xxx files which include -x on Message-ID: Revision: 6363 Author: nancy Date: 2013-08-01 13:48:45 -0600 (Thu, 01 Aug 2013) Log Message: ----------- found a few other mkmf_xxx files which include -x on the mkmf line; remove it to be consistent with the rest of the mkmf_xxx files in the system. (trying to run make as part of the mkmf command caused problems for a bash user who apparently didn't have his compiler modules set up right when running inside a c shell.) Modified Paths: -------------- DART/trunk/observations/Ameriflux/work/mkmf_level4_to_obs DART/trunk/observations/MADIS/work/mkmf_convert_madis_acars DART/trunk/observations/snow/work/mkmf_snow_to_obs -------------- next part -------------- Modified: DART/trunk/observations/Ameriflux/work/mkmf_level4_to_obs =================================================================== --- DART/trunk/observations/Ameriflux/work/mkmf_level4_to_obs 2013-08-01 19:37:08 UTC (rev 6362) +++ DART/trunk/observations/Ameriflux/work/mkmf_level4_to_obs 2013-08-01 19:48:45 UTC (rev 6363) @@ -7,7 +7,7 @@ # DART $Id$ ../../../mkmf/mkmf -p level4_to_obs -t ../../../mkmf/mkmf.template \ - -x -a "../../.." path_names_level4_to_obs + -a "../../.." path_names_level4_to_obs exit $status Modified: DART/trunk/observations/MADIS/work/mkmf_convert_madis_acars =================================================================== --- DART/trunk/observations/MADIS/work/mkmf_convert_madis_acars 2013-08-01 19:37:08 UTC (rev 6362) +++ DART/trunk/observations/MADIS/work/mkmf_convert_madis_acars 2013-08-01 19:48:45 UTC (rev 6363) @@ -7,7 +7,7 @@ # DART $Id$ ../../../mkmf/mkmf -p convert_madis_acars -t ../../../mkmf/mkmf.template \ - -x -a "../../.." path_names_convert_madis_acars + -a "../../.." path_names_convert_madis_acars exit $status Modified: DART/trunk/observations/snow/work/mkmf_snow_to_obs =================================================================== --- DART/trunk/observations/snow/work/mkmf_snow_to_obs 2013-08-01 19:37:08 UTC (rev 6362) +++ DART/trunk/observations/snow/work/mkmf_snow_to_obs 2013-08-01 19:48:45 UTC (rev 6363) @@ -7,7 +7,7 @@ # DART $Id$ ../../../mkmf/mkmf -p snow_to_obs -t ../../../mkmf/mkmf.template \ - -x -a "../../.." path_names_snow_to_obs + -a "../../.." path_names_snow_to_obs exit $status From nancy at ucar.edu Thu Aug 1 14:33:24 2013 From: nancy at ucar.edu (nancy at ucar.edu) Date: Thu, 01 Aug 2013 14:33:24 -0600 Subject: [Dart-dev] [6364] DART/trunk/mpi_utilities/fixsystem: add optional arguments to request the desired settings - works Message-ID: Revision: 6364 Author: nancy Date: 2013-08-01 14:33:23 -0600 (Thu, 01 Aug 2013) Log Message: ----------- add optional arguments to request the desired settings - works for gfortran or anything else. still fully backwards compatible; called with no args it will alternate between settings. added more comments to the code, changed the messages that are printed out to be possibly more user-friendly. Modified Paths: -------------- DART/trunk/mpi_utilities/fixsystem -------------- next part -------------- Modified: DART/trunk/mpi_utilities/fixsystem =================================================================== --- DART/trunk/mpi_utilities/fixsystem 2013-08-01 19:48:45 UTC (rev 6363) +++ DART/trunk/mpi_utilities/fixsystem 2013-08-01 20:33:23 UTC (rev 6364) @@ -6,59 +6,107 @@ # # $Id$ +# usage: fixsystem [ -gfortran | -not_gfortran ] +# +# in two DART source code files in this directory we depend on using +# the system() function to run a shell script and wait for the shell +# exit code, e.g.: rc = system("/bin/date") +# +# for all compilers, except gfortran, an interface block is required +# to define the integer return from the system function. however +# the gfortran compiler gives an error if this block is defined. +# this script tries to comment in and out this interface block by +# looking for a pair of specially formatted comment lines and +# commenting in (or out) all the lines between those comment +# delimiter lines. +# +# the original usage of this script was without any arguments. +# it swapped the state of the comment block; if it was commented out +# it removed the comment characters, or if the code was enabled +# it added comment chars. it still has this backwards-compatible +# behavior but now it also takes a single optional argument +# which must be one of: -gfortran or -not_gfortran +# it ensures the interface block is in the correct configuration +# for the given compiler. + for f in mpi_utilities_mod.f90 null_mpi_utilities_mod.f90 do -# save original copy for backup. should this test first and if the orig -# is still there, not do the copy? -if [[ ! -f ${f}.orig ]]; then - cp -p ${f} ${f}.orig -fi + # figure out what state the source file is in before we start + export bline=`fgrep SYSTEM_BLOCK_EDIT ${f} | grep START | head -n 1` + if [[ "`echo $bline | grep COMMENTED_OUT`" != "" ]]; then + #echo Is no interface block for system in ${f} + export before=out + elif [[ "`echo $bline | grep COMMENTED_IN`" != "" ]]; then + #echo Interface block is present in ${f} + export before=in + else + echo ${f} not found, or does not have the right comment string to + echo automatically change the system interface block via script. + echo Please restore original file from the subversion repository + echo and try again. + exit 1 + fi -# door 1: flip flop back and forth between commented in and out -# door 2: give this an argument to ask for it in or out explicitly -# door 3: look at an env var for what to set. (could be in mkmf.template) + if [[ $# == 0 ]]; then + # no args given, swap to the other configuration + if [[ $before == out ]]; then + export todo=in + elif [[ $before == in ]]; then + export todo=out + else + echo Internal error; should not happen. Contact DART support. + exit 1 + fi -# door 1: -export bline=`fgrep SYSTEM_BLOCK_EDIT ${f} | grep START` + elif [[ $# == 1 ]]; then + # single arg: must be either -gfortran or -not_gfortran + if [[ "$1" == -gfortran ]]; then + export todo=out + elif [[ "$1" == -not_gfortran ]]; then + export todo=in + else + echo unrecognized argument \"$1\" given to $0 + echo valid args are either \"-gfortran\" or \"-not_gfortran\" + exit 1 + fi -if [[ "`echo $bline | grep COMMENTED_OUT`" != "" ]]; then - echo Was no interface block for system in ${f} - export do=in -elif [[ "`echo $bline | grep COMMENTED_IN`" != "" ]]; then - echo Interface block was present in ${f} - export do=out -else - echo ${f} not found, or does not have the right comment string to - echo automatically change the system interface block via script. - echo Please restore original file from the subversion repository - echo and try again. -fi + else + # too many arguments, give an error message and exit + echo invalid usage, more than 1 argument given to $0 + echo only one of either \"-gfortran\" or \"-not_gfortran\" can be specified + exit 1 + fi + + # if we are already in the right state, loop to next file + if [[ $before == out && $todo == out ]]; then continue; fi + if [[ $before == in && $todo == in ]]; then continue; fi + + # we do have something do to -# in any case, here are the 4 possible command lines to use. + # save original copy for backup if one does not already exist. + if [[ ! -f ${f}.orig ]]; then + cp -p ${f} ${f}.orig + fi + + # removing comment chars, enabling interface block code + if [[ $todo == in ]]; then + echo Setting for a non-gfortran compiler in ${f} + mv ${f} tempfile + sed -e '/SYSTEM_BLOCK_EDIT START COMMENTED_OUT/,/SYSTEM_BLOCK_EDIT END COMMENTED_OUT/s/^!//' \ + -e '/$SYSTEM_BLOCK_EDIT [A-Z][A-Z]*$ COMMENTED_OUT/s//\1 COMMENTED_IN/' tempfile > ${f} + fi + + # adding comment chars, disabling interface block code + if [[ $todo == out ]]; then + echo Setting for gfortran compiler in ${f} + mv ${f} tempfile + sed -e '/SYSTEM_BLOCK_EDIT START COMMENTED_IN/,/SYSTEM_BLOCK_EDIT END COMMENTED_IN/s/^/!/' \ + -e '/$SYSTEM_BLOCK_EDIT [A-Z][A-Z]*$ COMMENTED_IN/s//\1 COMMENTED_OUT/' tempfile > ${f} + fi -# removing comments, code enabled -if [[ $do == in ]]; then - echo removing comments, enabling interface block in ${f} - mv ${f} tempfile - sed -e '/SYSTEM_BLOCK_EDIT START COMMENTED_OUT/,/SYSTEM_BLOCK_EDIT END COMMENTED_OUT/s/^!//' -e '/$SYSTEM_BLOCK_EDIT [A-Z][A-Z]*$ COMMENTED_OUT/s//\1 COMMENTED_IN/' tempfile > ${f} -fi + \rm -f tempfile -# adding comments, code disabled -if [[ $do == out ]]; then - echo adding comment chars, disabling interface block in ${f} - mv ${f} tempfile - sed -e '/SYSTEM_BLOCK_EDIT START COMMENTED_IN/,/SYSTEM_BLOCK_EDIT END COMMENTED_IN/s/^/!/' -e '/$SYSTEM_BLOCK_EDIT [A-Z][A-Z]*$ COMMENTED_IN/s//\1 COMMENTED_OUT/' tempfile > ${f} -fi - -# for testing only -## trying to remove comments not there; should be noop -#sed -e '/SYSTEM_BLOCK_EDIT START COMMENTED_OUT/,/SYSTEM_BLOCK_EDIT END COMMENTED_OUT/s/^!//' -e '/$SYSTEM_BLOCK_EDIT [A-Z][A-Z]*$ COMMENTED_OUT/s//\1 COMMENTED_IN/' new > new3 -# -## trying to add comments already there; should be noop -#sed -e '/SYSTEM_BLOCK_EDIT START COMMENTED_IN/,/SYSTEM_BLOCK_EDIT END COMMENTED_IN/s/^/!/' -e '/$SYSTEM_BLOCK_EDIT [A-Z][A-Z]*$ COMMENTED_IN/s//\1 COMMENTED_OUT/' new2 > new4 -# - done exit 0 From nancy at ucar.edu Thu Aug 1 16:48:02 2013 From: nancy at ucar.edu (nancy at ucar.edu) Date: Thu, 01 Aug 2013 16:48:02 -0600 Subject: [Dart-dev] [6368] DART/trunk/doc: move these gif images related to async run modes Message-ID: Revision: 6368 Author: nancy Date: 2013-08-01 16:48:01 -0600 (Thu, 01 Aug 2013) Log Message: ----------- move these gif images related to async run modes from the html dir to the images dir. update the single html page that references them. Modified Paths: -------------- DART/trunk/doc/html/filter_async_modes.html Added Paths: ----------- DART/trunk/doc/images/async0.gif DART/trunk/doc/images/async2_v1.gif DART/trunk/doc/images/async2_v2.gif DART/trunk/doc/images/async2_wfiles.gif DART/trunk/doc/images/async2a.gif DART/trunk/doc/images/async4.gif DART/trunk/doc/images/async4_wfiles.gif -------------- next part -------------- Modified: DART/trunk/doc/html/filter_async_modes.html =================================================================== --- DART/trunk/doc/html/filter_async_modes.html 2013-08-01 22:12:59 UTC (rev 6367) +++ DART/trunk/doc/html/filter_async_modes.html 2013-08-01 22:48:01 UTC (rev 6368) @@ -29,7 +29,7 @@
Simplest case, async=0:
- +
This is a single MPI executable, with each call to the model being simply a subroutine call from each MPI task. @@ -43,7 +43,7 @@
Parallel advance, async=2:
- +
The filter executable is one MPI program, and the model is a single, sequential executable. Each MPI task uses the @@ -59,13 +59,13 @@
Parallel advance, async=2:
- +

Parallel advance, async=2, second version:
- +

@@ -73,7 +73,7 @@ between filter and the model thru intermediate files. IC are 'initial condition' files, UD are 'updated' files.
- +

@@ -81,7 +81,7 @@
Parallel model advance, async=4:
- +
The filter executable is one MPI program, and the model is also an MPI program. The filter executable communicates with the @@ -98,7 +98,7 @@ between filter and the model thru intermediate files. IC are 'initial condition' files, UD are 'updated' files.
- +

Copied: DART/trunk/doc/images/async0.gif (from rev 6336, DART/trunk/doc/html/async0.gif) =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/async2_v1.gif (from rev 6336, DART/trunk/doc/html/async2_v1.gif) =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/async2_v2.gif (from rev 6336, DART/trunk/doc/html/async2_v2.gif) =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/async2_wfiles.gif (from rev 6336, DART/trunk/doc/html/async2_wfiles.gif) =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/async2a.gif (from rev 6336, DART/trunk/doc/html/async2a.gif) =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/async4.gif (from rev 6336, DART/trunk/doc/html/async4.gif) =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/async4_wfiles.gif (from rev 6336, DART/trunk/doc/html/async4_wfiles.gif) =================================================================== (Binary files differ) From nancy at ucar.edu Thu Aug 1 16:49:55 2013 From: nancy at ucar.edu (nancy at ucar.edu) Date: Thu, 01 Aug 2013 16:49:55 -0600 Subject: [Dart-dev] [6369] DART/trunk: finish the commit of the moved async gifs; Message-ID: Revision: 6369 Author: nancy Date: 2013-08-01 16:49:55 -0600 (Thu, 01 Aug 2013) Log Message: ----------- finish the commit of the moved async gifs; move the comm pattern files to the images dir and update the single html page that references them. Modified Paths: -------------- DART/trunk/ensemble_manager/ensemble_manager_mod.html Added Paths: ----------- DART/trunk/doc/images/comm_pattern512.png DART/trunk/doc/images/comm_pattern96.png Removed Paths: ------------- DART/trunk/doc/html/async0.gif DART/trunk/doc/html/async2_v1.gif DART/trunk/doc/html/async2_v2.gif DART/trunk/doc/html/async2_wfiles.gif DART/trunk/doc/html/async2a.gif DART/trunk/doc/html/async4.gif DART/trunk/doc/html/async4_wfiles.gif DART/trunk/doc/html/comm_pattern512.png DART/trunk/doc/html/comm_pattern96.png -------------- next part -------------- Deleted: DART/trunk/doc/html/async0.gif =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/async2_v1.gif =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/async2_v2.gif =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/async2_wfiles.gif =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/async2a.gif =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/async4.gif =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/async4_wfiles.gif =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/comm_pattern512.png =================================================================== (Binary files differ) Deleted: DART/trunk/doc/html/comm_pattern96.png =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/comm_pattern512.png (from rev 6336, DART/trunk/doc/html/comm_pattern512.png) =================================================================== (Binary files differ) Copied: DART/trunk/doc/images/comm_pattern96.png (from rev 6336, DART/trunk/doc/html/comm_pattern96.png) =================================================================== (Binary files differ) Modified: DART/trunk/ensemble_manager/ensemble_manager_mod.html =================================================================== --- DART/trunk/ensemble_manager/ensemble_manager_mod.html 2013-08-01 22:48:01 UTC (rev 6368) +++ DART/trunk/ensemble_manager/ensemble_manager_mod.html 2013-08-01 22:49:55 UTC (rev 6369) @@ -164,8 +164,8 @@


	Instructions for viewing the rank histogram with ncview.

	Instructions for viewing the rank histogram with Matlab.

Quickstart for CESM/DART

Get CESM

Required patches to CESM1.1.1

DART

EVERYTHING BELOW HERE PERTAINS ONLY TO POP AND NEEDS UPDATING

EVERYTHING BELOW HERE PERTAINS ONLY TO POP AND NEEDS UPDATING

EVERYTHING BELOW HERE PERTAINS ONLY TO POP AND NEEDS UPDATING

CESM1.0 POP2

DART Lanai Differences from Kodiak Release Notes

New Models or Changes to Existing Models

Changes to Core DART routines

New or changed Forward Operators

Observation Converters

New or updated DART Diagnostics

Tutorial, Scripting, Setup, Builds

Terms of Use

DART Lanai Release Notes

Dart Overview

Getting Started

What's Required

What's nice to have

Types of input

Document conventions

Installation

Requirements: an F90 Compiler

Requirements: the netCDF library

Downloading the distribution.

Customizing the build scripts -- Overview.

Building and Customizing the 'mkmf.template' file