[Dart-dev] [6495] DART/trunk/ensemble_manager/ensemble_manager_mod.html: major updating to make the docs consistent with the

nancy at ucar.edu nancy at ucar.edu
Wed Sep 25 16:16:34 MDT 2013


Revision: 6495
Author:   nancy
Date:     2013-09-25 16:16:33 -0600 (Wed, 25 Sep 2013)
Log Message:
-----------
major updating to make the docs consistent with the
current state of the ensemble manager code.

Modified Paths:
--------------
    DART/trunk/ensemble_manager/ensemble_manager_mod.html

-------------- next part --------------
Modified: DART/trunk/ensemble_manager/ensemble_manager_mod.html
===================================================================
--- DART/trunk/ensemble_manager/ensemble_manager_mod.html	2013-09-25 21:34:23 UTC (rev 6494)
+++ DART/trunk/ensemble_manager/ensemble_manager_mod.html	2013-09-25 22:16:33 UTC (rev 6495)
@@ -52,6 +52,34 @@
 and quality control and ensembles of forward observation operator error 
 status.
 </P>
+<P>
+The ensemble manager interacts strongly with the multiple process capability
+of the Message Passing Interface (MPI) libraries.  It is used to partition
+the data so each MPI process stores only a subset of the copies and variables,
+dividing the data as evenly as possible across the processes.  At no time
+during the execution does any one process have to store the entire dataset
+for all ensemble members (unless running in serial mode without MPI, or if
+running with 1 MPI task).
+</P>
+<P>
+The ensemble manager is set of general purpose data management routines.
+For run-time efficiency, the derived type information is not marked private which
+means other modules can directly manipulate the data arrays.
+However it means much care must be taken to access the most recently 
+updated representation of the data, either the copies or variables arrays.
+</P>
+<P>
+A set of sanity check routines have been added to track the last modified
+version of the data: the copies array or the vars array.  Before directly
+reading or writing these arrays call one of the 'prepare' routines to
+indicate what kind of data access you are about to make. If the most recently
+updated data is not as expected an error message will occur.  After the
+direct access if the following operations detect that the data they are
+operating on is not the most recently updated they will print an error message.
+Routines inside the ensemble manager that alter the copies or vars will set
+the state automatically so these routines are only necessary to call if you 
+are directly accessing the copies or vars arrays from outside the ensemble manager.
+</P>
 
 <!--==================================================================-->
 <!--=================== DESCRIPTION OF A NAMELIST  ===================-->
@@ -133,7 +161,7 @@
 <TR><TD>layout</TD>
     <TD>integer</TD>
     <TD>Determines the process (pe) layout on MPI tasks.
-     1 is pe = MPI task. 2 is a round-robin layout around the nodes. Layout 2
+     1 is PE = MPI task. 2 is a round-robin layout around the nodes. Layout 2
      results in a more even usage of memory across nodes. This may allow you to run
      with a larger state vector without hitting the memory limit of the node.
      It may give a slight (5%) increase in performance, but this is machine dependent.
@@ -190,8 +218,10 @@
 
 <div>
 <table width="100%" summary='communication patterns'><tr>
-<td><a href="../doc/images/comm_pattern96.png"><img src="../doc/images/comm_pattern96.png" width="400"></a>
-<td><a href="../doc/images/comm_pattern512.png"><img src="../doc/images/comm_pattern512.png" width="400"></a>
+<td><a href="../doc/images/comm_pattern96.png">
+<img src="../doc/images/comm_pattern96.png" width="400" alt="communication pattern"></a>
+<td><a href="../doc/images/comm_pattern512.png">
+<img src="../doc/images/comm_pattern512.png" width="400" alt="communication pattern 2"></a>
 </table>
 </div>
 
@@ -209,6 +239,7 @@
 time_manager_mod
 random_seq_mod
 mpi_utilities_mod
+sort_mod
 </PRE>
 
 <!--==================================================================-->
@@ -222,27 +253,38 @@
 <div class="top">[<a href="#">top</a>]</div><hr />
 <H2>PUBLIC INTERFACES</H2>
 
-<TABLE>
+<TABLE summary='public routine list'>
 <TR><TD><em class=call>use ensemble_manager_mod, only : </em></TD>
-                   <TD><A HREF="#all_copies_to_all_vars">all_copies_to_all_vars</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#all_vars_to_all_copies">all_vars_to_all_copies</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#compute_copy_mean">compute_copy_mean</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#compute_copy_mean_sd">compute_copy_mean_sd</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#compute_copy_mean_var">compute_copy_mean_var</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#duplicate_ens">duplicate_ens</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#end_ensemble_manager">end_ensemble_manager</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#ensemble_type">ensemble_type</A></TD></TR>
+                   <TD><A HREF="#init_ensemble_manager">init_ensemble_manager</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#read_ensemble_restart">read_ensemble_restart</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#write_ensemble_restart">write_ensemble_restart</A></TD></TR>
 <TR><TD>&nbsp;</TD><TD><A HREF="#get_copy">get_copy</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#put_copy">put_copy</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#broadcast_copy">broadcast_copy</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#set_ensemble_time">set_ensemble_time</A></TD></TR>
 <TR><TD>&nbsp;</TD><TD><A HREF="#get_ensemble_time">get_ensemble_time</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#end_ensemble_manager">end_ensemble_manager</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#duplicate_ens">duplicate_ens</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#get_my_num_copies">get_my_num_copies</A></TD></TR>
 <TR><TD>&nbsp;</TD><TD><A HREF="#get_my_copies">get_my_copies</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#get_my_num_copies">get_my_num_copies</A></TD></TR>
 <TR><TD>&nbsp;</TD><TD><A HREF="#get_my_num_vars">get_my_num_vars</A></TD></TR>
 <TR><TD>&nbsp;</TD><TD><A HREF="#get_my_vars">get_my_vars</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#get_copy_owner_index">get_copy_owner_index</A></TD></TR>
 <TR><TD>&nbsp;</TD><TD><A HREF="#get_var_owner_index">get_var_owner_index</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#init_ensemble_manager">init_ensemble_manager</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#put_copy">put_copy</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#read_ensemble_restart">read_ensemble_restart</A></TD></TR>
-<TR><TD>&nbsp;</TD><TD><A HREF="#write_ensemble_restart">write_ensemble_restart</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#all_vars_to_all_copies">all_vars_to_all_copies</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#all_copies_to_all_vars">all_copies_to_all_vars</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#compute_copy_mean">compute_copy_mean</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#compute_copy_mean_sd">compute_copy_mean_sd</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#compute_copy_mean_var">compute_copy_mean_var</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#prepare_to_write_to_vars">prepare_to_write_to_vars</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#prepare_to_write_to_copies">prepare_to_write_to_copies</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#prepare_to_read_from_vars">prepare_to_read_from_vars</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#prepare_to_read_from_copies">prepare_to_read_from_copies</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#prepare_to_update_vars">prepare_to_update_vars</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#prepare_to_update_copies">prepare_to_update_copies</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#print_ens_handle">print_ens_handle</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#map_pe_to_task">map_pe_to_task</A></TD></TR>
+<TR><TD>&nbsp;</TD><TD><A HREF="#map_task_to_pe">map_task_to_pe</A></TD></TR>
 </TABLE>
 
 <P>
@@ -253,23 +295,32 @@
 
 <!--=================== DESCRIPTION OF A LOCAL TYPE ==================-->
 
-<!--kdr no private in code -->
 <A NAME="ensemble_type"></A>
 <BR>
 <div class=type>
 <pre>
 <em class=call>type ensemble_type</em>
-   !!! private
+   !DIRECT ACCESS INTO STORAGE IS ALLOWED; BE CAREFUL
    integer :: num_copies
    integer :: num_vars
    integer :: my_num_copies
    integer :: my_num_vars
    integer, pointer :: my_copies(:)
    integer, pointer :: my_vars(:)
-   real(r8), pointer :: copies(:, :)
-   real(r8), pointer :: vars(:, :)
+   ! Storage in next line is to be used when each PE has all copies of subset of vars
+   real(r8), pointer :: copies(:, :)  ! Dimensioned (num_copies, my_num_vars)
+   ! Storage on next line is used when each PE has subset of copies of all vars
+   real(r8), pointer :: vars(:, :)    ! Dimensioned (num_vars, my_num_copies)
+   ! Time is only related to var complete
    type(time_type), pointer :: time(:)
    integer :: distribution_type
+   integer :: valid     ! copies modified last, vars modified last, both same
+   integer :: id_num
+   integer, allocatable :: task_to_pe_list(:) ! List of tasks
+   integer, allocatable :: pe_to_task_list(:) ! List of tasks
+   ! Flexible my_pe, layout_type which allows different task layouts for different ensemble handles
+   integer :: my_pe
+   integer :: layout_type
 end type ensemble_type
 </pre>
 </div>
@@ -283,35 +334,69 @@
 used throughout DART. 
 </P>
 
-<TABLE border=0 cellpadding=3 width=100%>
-<TR><TH align=left>Component      </TH>
-    <TH align=left>Description    </TH></TR>
-<TR><TD valign=top> num_copies </TD>
+<TABLE border=0 cellpadding=3 width=100% summary='derived type description'>
+<THEAD align=left>
+<TR><TH> Component   </TH>
+    <TH> Description </TH> </TR>
+</THEAD>
+
+<TBODY valign=top>
+
+<TR><TD> num_copies </TD>
     <TD> Global number of copies of the vector.  </TD></TR>
-<TR><TD valign=top> num_vars </TD>
+<TR><TD> num_vars </TD>
     <TD> Global number of elements (variables) in the vector.  </TD></TR>
-<TR><TD valign=top> my_num_copies </TD>
+<TR><TD> my_num_copies </TD>
     <TD> Number of copies stored by this process.  </TD></TR>
-<TR><TD valign=top> my_num_vars </TD>
+<TR><TD> my_num_vars </TD>
     <TD> Number of variables stored by this process.  </TD></TR>
-<TR><TD valign=top> my_copies </TD>
+<TR><TD> my_copies </TD>
     <TD> Dimensioned to size my_num_copies. Contains a list of the global
          indices of copies stored by this process.  </TD></TR>
-<TR><TD valign=top> my_vars </TD>
+<TR><TD> my_vars </TD>
     <TD> Dimensioned to size my_num_vars. Contains a list of the global
          indices of variables stored by this process.  </TD></TR>
-<TR><TD valign=top> copies </TD>
+<TR><TD> copies </TD>
     <TD> Dimensioned (num_copies, my_num_vars). Storage for all copies
          of variables stored by this process.  </TD></TR>
-<TR><TD valign=top> vars </TD>
+<TR><TD> vars </TD>
     <TD> Dimensioned (num_vars, my_num_copies). Storage for all variables
          of copies stored by this process.  </TD></TR>
-<TR><TD valign=top> time </TD>
+<TR><TD> time </TD>
     <TD> Dimensioned my_num_copies. A time_type that stores time associated
          with a given copy of the vector.  </TD></TR>
-<TR><TD valign=top> distribution_type </TD>
+<TR><TD> distribution_type </TD>
     <TD>Does nothing at present. Can be used for future releases to control
          the layout of different copies and variables in storage.  </TD></TR>
+<TR><TD> valid </TD>
+    <TD> Flag to track whether the copies array has the most recently updated
+         data, the vars array is most recently modified, or if both the arrays
+         have identical data, like after a transpose.  </TD></TR>
+<TR><TD> id_num </TD>
+    <TD> Internal number unique to each ensemble handle, used for
+         debugging purposes.  </TD></TR>
+<TR><TD> task_to_pe_list </TD>
+    <TD> Mapping from MPI task number to logical Processing Element (PE) number.  
+         Enables different assignment of MPI tasks to PEs.  If the number of MPI 
+         tasks is larger than the number of copies of the vector, when the ensemble 
+         is var complete then the first N MPI tasks have allocated 'vars' arrays
+         and the remaining ones do not.  Assigning the MPI tasks round-robin 
+         to multi-processor nodes can make the memory usage more uniform across 
+         nodes, which may allow more MPI tasks per node than the standard layout.  </TD></TR>
+<TR><TD> pe_to_task_list </TD>
+    <TD> Logical PE to MPI task mapping.  See above for more description.
+        </TD></TR>
+<TR><TD> my_pe </TD>
+    <TD> The logical PE number for the MPI task.
+        </TD></TR>
+<TR><TD> layout_type </TD>
+    <TD> Controls the mapping type between MPI tasks and PEs.  Currently type 1
+         is the standard layout (one-to-one mapping) and type 2 is a round-robin
+         mapping where each node gets a task in turn before assigning a second
+         task to each node, until all tasks are assigned.
+        </TD></TR>
+
+</TBODY>
 </TABLE>
 
 </div>
@@ -322,13 +407,15 @@
 <A NAME="init_ensemble_manager"></A>
 <br>
 <div class=routine>
-<em class=call> call init_ensemble_manager(ens_handle, num_copies, 
-   num_vars <em class=optionalcode>[,&nbsp;distribution_type_in]</em>) </em>
+<em class=call> call init_ensemble_manager(ens_handle, num_copies, num_vars
+<em class=optionalcode>[,&nbsp;distribution_type_in]</em>
+<em class=optionalcode>[,&nbsp;layout_type]</em>) </em>
 <pre>
 type(ensemble_type), intent(out) :: <em class=code>ens_handle</em>
-integer, intent(in)              :: <em class=code>num_copies</em>
-integer, intent(in)              :: <em class=code>num_vars</em>
-integer, optional, intent(in)    :: <em class=optionalcode>distribution_type_in</em>
+integer,             intent(in)  :: <em class=code>num_copies</em>
+integer,             intent(in)  :: <em class=code>num_vars</em>
+integer, optional,   intent(in)  :: <em class=optionalcode>distribution_type_in</em>
+integer, optional,   intent(in)  :: <em class=optionalcode>layout_type</em>
 </pre>
 </div>
 
@@ -347,8 +434,14 @@
     <TD>Number of copies of vector.</TD></TR>
 <TR><TD valign=top><em class=code>num_vars</em></TD>
     <TD>Number of variables in the vector.</TD></TR>
-<TR><TD valign=top><em class=optionalcode>distribution_type_in&nbsp;&nbsp;&nbsp;</em></TD>
-    <TD>Controls layout of storage on pe's. Currently only option 1 is supported.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>distribution_type_in</em></TD>
+    <TD>Controls layout of storage on PEs. Currently only option 1 is supported.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>layout_type</em></TD>
+    <TD>Controls layout of MPI tasks on PEs.  Type 1 is the default, where MPI tasks
+        are assigned to PEs on a one-to-one basis.  Type 2 is a round-robin assignment
+        where each node gets one task before the nodes are assigned a second task.
+        If running with more MPI tasks than <em class=code>num_copies</em>, this can
+        result in a more uniform usage of memory across the nodes.  </TD></TR>
 </TABLE>
 
 </div>
@@ -356,14 +449,21 @@
 
 <!--===================== DESCRIPTION OF A ROUTINE =====================-->
 
-<A NAME="get_var_owner_index"></A>
+<A NAME="read_ensemble_restart"></A>
 <br>
 <div class=routine>
-<em class=call> call get_var_owner_index(var_number, owner, owners_index) </em>
+<em class=call> call read_ensemble_restart(ens_handle, start_copy, end_copy, 
+   start_from_restart, file_name
+<em class=optionalcode>[,&nbsp;init_time]</em>
+<em class=optionalcode>[,&nbsp;force_single_file]</em>) </em>
 <pre>
-integer, intent(in)  :: <em class=code>var_number</em>
-integer, intent(out) :: <em class=code>owner</em>
-integer, intent(out) :: <em class=code>owners_index</em>
+type(ensemble_type),       intent(inout) :: <em class=code>ens_handle</em>
+integer,                   intent(in)    :: <em class=code>start_copy</em>
+integer,                   intent(in)    :: <em class=code>end_copy</em>
+logical,                   intent(in)    :: <em class=code>start_from_restart</em>
+character(len=*),          intent(in)    :: <em class=code>file_name</em>
+type(time_type), optional, intent(in)    :: <em class=optionalcode>init_time</em>
+logical, optional,         intent(in)    :: <em class=optionalcode>force_single_file</em>
 </pre>
 </div>
 
@@ -371,19 +471,41 @@
 <!-- Description -->
 
 <P>
-Given the global index of a variable in the vector, returns the process that
-stores this variable when all copies of a subset of variables are stored and
-the local storage index for this variable on that process.
+Read in a set of copies of a vector from file file_name. The copies read are
+place into global copies start_copy:end_copy in the ens_handle. 
+If start_from_restart is false, then only a single copy of the vector is
+read from the file and then it is perturbed using routines in assim_model_mod
+to generate the required number of copies. The read can be from a single file
+that contains all needed copies or from a different file for each copy. This
+choice is controlled by the namelist entry single_restart_file_in. However, the
+optional argument force_single_file forces the read to be from a single file
+if it is present and true. This is used for ensembles that contain the
+inflation values for state space inflation. If multiple files are to be read,
+the file names are generated by appending integers to the input file_name.
+If the input is a single file all reads are done sequentially by process 0 
+and then shipped to the PE that stores that copy.
+If the input is multiple files each MPI task reads the copies it stores
+directly and independently.
 </P>
 
 <TABLE width=100% border=0 summary="" cellpadding=3>
-<TR><TD valign=top><em class=code>var_number</em></TD>
-    <TD>Global index of a variable in the vector from an ensemble.</TD></TR>
-<TR><TD valign=top><em class=code>owner</em></TD>
-    <TD>Process (0 to num_processes) that stores this variable when each has all 
-         copies of subset of variables.</TD></TR>
-<TR><TD valign=top><em class=code>owners_index&nbsp;&nbsp;&nbsp;</em></TD>
-    <TD>Local storage index for this variable on the owning process.</TD></TR>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle of ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>start_copy&nbsp;&nbsp;</em></TD>
+    <TD>Global index of first of continguous set of copies to be read.</TD></TR>
+<TR><TD valign=top><em class=code>end_copy&nbsp;&nbsp;</em></TD>
+    <TD>Global index of last of contiguous set of copies to be read, 
+       copies(start_copy:end_copy).</TD></TR>
+<TR><TD valign=top><em class=code>start_from_restart&nbsp;&nbsp;</em></TD>
+    <TD>If true, read all copies from file. If false, read one copy and perturb
+ to get required number.</TD></TR>
+<TR><TD valign=top><em class=code>file_name&nbsp;&nbsp;</em></TD>
+    <TD>Name of file from which to read.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>init_time&nbsp;&nbsp;</em></TD>
+    <TD>If present, set time of all copies read to this value.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>force_single_file&nbsp;&nbsp;</em></TD>
+    <TD>If present and true, force the read to be from a single file which 
+      contains all copies.</TD></TR>
 </TABLE>
 
 </div>
@@ -391,6 +513,388 @@
 
 <!--===================== DESCRIPTION OF A ROUTINE =====================-->
 
+<A NAME="write_ensemble_restart"></A>
+<br>
+<div class=routine>
+<em class=call> call write_ensemble_restart(ens_handle, file_name, start_copy, 
+ end_copy <em class=optionalcode>[,&nbsp;force_single_file]</em>) </em>
+<pre>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+character(len=*),    intent(in)    :: <em class=code>file_name</em>
+integer,             intent(in)    :: <em class=code>start_copy</em>
+integer,             intent(in)    :: <em class=code>end_copy</em>
+logical, optional,   intent(in)    :: <em class=optionalcode>force_single_file</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Writes a set of copies of a vector to file file_name. The copies written are
+from global copies start_copy:end_copy in the ens_handle. 
+The write can be to a single file or to a different file for each copy. This
+choice is controlled by the namelist entry single_restart_file_out. However, the
+optional argument force_single_file forces the write to be to a single file
+if it is present and true. This is used for ensembles that contain the
+inflation values for state space inflation. If multiple files are to be written,
+the file names are generated by appending integers to the input file_name.
+If the output is a single file all copies are shipped from the PE that
+stores that copy to process 0, and then written out sequentially.
+If the output is to multiple files each MPI task writes the copies it stores
+directly and independently.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle of ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>file_name&nbsp;&nbsp;</em></TD>
+    <TD>Name of file from which to read.</TD></TR>
+<TR><TD valign=top><em class=code>start_copy&nbsp;&nbsp;</em></TD>
+    <TD>Global index of first of continguous set of copies to be written.</TD></TR>
+<TR><TD valign=top><em class=code>end_copy&nbsp;&nbsp;</em></TD>
+    <TD>Global index of last of contiguous set of copies to be written, 
+          copies(start_copy:end_copy).</TD></TR>
+<TR><TD valign=top><em class=optionalcode>force_single_file&nbsp;&nbsp;</em></TD>
+    <TD>If present and true, force the write to be to a single file which 
+     contains all copies.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="get_copy"></A>
+<br>
+<div class=routine>
+<em class=call> call get_copy(receiving_pe, ens_handle, copy, vars
+<em class=optionalcode>[,&nbsp;mtime]</em>) </em>
+<pre>
+integer,                   intent(in)  :: <em class=code>receiving_pe</em>
+type(ensemble_type),       intent(in)  :: <em class=code>ens_handle</em>
+integer,                   intent(in)  :: <em class=code>copy</em>
+real(r8), dimension(:),    intent(out) :: <em class=code>vars</em>
+type(time_type), optional, intent(out) :: <em class=optionalcode>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Retrieves a copy of the state vector, indexed by the global index copy. The
+process that is to receive the copy is receiving_pe and the copy is returned
+in the one dimensional array vars. The time of the copy is also returned if
+mtime is present. This is generally used for operations, like IO, that require
+a single processor to do things with the entire state vector.
+Data is only returned in vars on the receiving PE; vars on all other PEs is unset.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>receiving_pe&nbsp;&nbsp;</em></TD>
+    <TD>This process ends up with the requested copy of the state 
+         vector.</TD></TR>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy&nbsp;&nbsp;</em></TD>
+    <TD>The global index of the copy of the state vector that is to be 
+         retrieved.</TD></TR>
+<TR><TD valign=top><em class=code>vars&nbsp;&nbsp;</em></TD>
+    <TD>One dimensional array in which the requested copy of the state vector 
+         is returned. Data is only returned in vars on the receiving PE;
+         vars on all other PEs is unset.  </TD></TR>
+<TR><TD valign=top><em class=optionalcode>mtime&nbsp;&nbsp;</em></TD>
+    <TD>If present returns the time of the requested copy.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="put_copy"></A>
+<br>
+<div class=routine>
+<em class=call> call put_copy(sending_pe, ens_handle, copy, vars
+<em class=optionalcode>[,&nbsp;mtime]</em>) </em>
+<pre>
+integer,                   intent(in)    :: <em class=code>sending_pe</em>
+type(ensemble_type),       intent(inout) :: <em class=code>ens_handle</em>
+integer,                   intent(in)    :: <em class=code>copy</em>
+real(r8), dimension(:),    intent(in)    :: <em class=code>vars</em>
+type(time_type), optional, intent(in)    :: <em class=optionalcode>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Sends a state vector, in vars, from the given process to the process 
+storing the global index copy. 
+The time of the copy is also sent if mtime is present. 
+This is generally used for operations, like IO, that require
+a single processor to do things with the entire state vector. For instance,
+if a single process reads in a state vector, it can be shipped to the storing
+process by this subroutine.  Only the data in vars on the sending PE
+is processed; vars on all other PEs is ignored.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>sending_pe&nbsp;&nbsp;</em></TD>
+    <TD>This process sends the copy of the state vector.</TD></TR>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy&nbsp;&nbsp;</em></TD>
+    <TD>The global index of the copy of the state vector that is to be sent. 
+</TD></TR>
+<TR><TD valign=top><em class=code>vars&nbsp;&nbsp;</em></TD>
+    <TD>One dimensional array in which the requested copy of the state vector 
+        is located. Only the data in vars on the sending PE
+        is processed; vars on all other PEs is ignored.  </TD></TR>
+<TR><TD valign=top><em class=optionalcode>mtime&nbsp;&nbsp;</em></TD>
+    <TD>If present send the time of the copy.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="broadcast_copy"></A>
+<br>
+<div class=routine>
+<em class=call> call broadcast_copy(ens_handle, copy, arraydata)</em>
+<pre>
+type(ensemble_type),    intent(in)   :: <em class=code>ens_handle</em>
+integer,                intent(in)   :: <em class=code>copy</em>
+real(r8), dimension(:), intent(out)  :: <em class=code>arraydata</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Finds which PE has the global index copy and broadcasts
+that copy to all PEs.  <em class=code>arraydata</em> is an output on
+all PEs, even on the PE which is the owner if it is separate
+storage from the vars array in the ensemble handle.
+This is a collective routine, which means it must be called by
+all processes in the job.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy&nbsp;&nbsp;</em></TD>
+    <TD>The global index of the copy of the state vector that is to be sent. 
+</TD></TR>
+<TR><TD valign=top><em class=code>arraydata&nbsp;&nbsp;</em></TD>
+    <TD>One dimensional array into which the requested copy of the state vector 
+        will be copied on all PEs, including the sending PE.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="set_ensemble_time"></A>
+<br>
+<div class=routine>
+<em class=call> call set_ensemble_time(ens_handle, indx, mtime) </em>
+<pre>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+integer,             intent(in)    :: <em class=code>indx</em>
+type(time_type),     intent(in)    :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Set the time of a copy to the given value.  <em class=code>indx</em> in 
+this case is the local copy number for a specific task.  
+<!-- Contrast this with 
+<a href="#set_copy_time">set_copy_time()</a> which uses global copy numbers. -->
+<a href="#get_copy_owner_index">get_copy_owner_index()</a>
+can be called to see if you are the owning task for a given global copy number,
+and to get the local index number for that copy.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>indx&nbsp;&nbsp;</em></TD>
+    <TD>The local index of the copy of the state vector that is to be set. 
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime&nbsp;&nbsp;</em></TD>
+    <TD>The time to set for this copy. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="get_ensemble_time"></A>
+<br>
+<div class=routine>
+<em class=call> call get_ensemble_time(ens_handle, indx, mtime) </em>
+<pre>
+type(ensemble_type), intent(in)   :: <em class=code>ens_handle</em>
+integer,             intent(in)   :: <em class=code>indx</em>
+type(time_type),     intent(out)  :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Get the time associated with a copy.  <em class=code>indx</em> in 
+this case is the local copy number for a specific task.  
+<a href="#get_copy_owner_index">get_copy_owner_index()</a>
+can be called to see if you are the owning task for a given global copy number,
+and to get the local index number for that copy.
+<!-- Contrast this with 
+<a href="#get_copy_time">get_copy_time()</a> which uses global copy numbers. -->
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>indx&nbsp;&nbsp;</em></TD>
+    <TD>The local index of the copy to retrieve the time from.
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime&nbsp;&nbsp;</em></TD>
+    <TD>The returned time value. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--
+
+this is probably a good idea, but the code hasn't been tested yet
+and it's close to a release, so leave it out for now.
+
+....===================== DESCRIPTION OF A ROUTINE =====================...
+
+<A NAME="set_copy_time"></A>
+<br>
+<div class=routine>
+<em class=call> call set_copy_time(ens_handle, copy, mtime) </em>
+<pre>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+
+type(time_type),     intent(in)    :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+< ! - - Description - - >
+
+<P>
+Set the time of a copy to the given value.  <em class=code>copy</em> in 
+this case is the global copy number.  Contrast this with 
+<a href="#set_ensemble_time">set_ensemble_time()</a> which uses local 
+copy indices.  
+This routine must be called by all processes and with the same mtime
+but only the process that stores the requested copy will do the update.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy&nbsp;&nbsp;</em></TD>
+    <TD>The global number of the copy that is to be set. 
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime&nbsp;&nbsp;</em></TD>
+    <TD>The time to set for this copy. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+....===================== DESCRIPTION OF A ROUTINE =====================...
+
+not sure if this is such a good idea.  only the task that actually
+stores the copy will return it, unless we do some communication.
+should we make it communicate?
+
+<A NAME="get_copy_time"></A>
+<br>
+<div class=routine>
+<em class=call> call get_copy_time(ens_handle, copy, mtime) </em>
+<pre>
+type(ensemble_type), intent(in)   :: <em class=code>ens_handle</em>
+integer,             intent(in)   :: <em class=code>copy</em>
+type(time_type),     intent(out)  :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+< ! - - Description - - >
+
+<P>
+Get the time associated with a copy.  <em class=code>copy</em> in 
+this case is the global copy number.  Contrast this with 
+<a href="#get_ensemble_time">get_ensemble_time()</a> which uses local
+copy indices.
+This routine can be called by all processes and only the
+process that stores the requested copy will return a value so it
+must be combined with a call to get_copy_owner_index() to see if
+you are the owner.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy&nbsp;&nbsp;</em></TD>
+    <TD>The global index of the copy to retrieve the time from.
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime&nbsp;&nbsp;</em></TD>
+    <TD>The returned time value. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+-->
+
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="end_ensemble_manager"></A>
+<br>
+<div class=routine>
+<em class=call> call end_ensemble_manager(ens_handle) </em>
+<pre>
+type(ensemble_type), intent(in)  :: <em class=code>ens_handle</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Frees up storage associated with an ensemble.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle&nbsp;&nbsp;</em></TD>
+    <TD>Handle for an ensemble.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
 <A NAME="duplicate_ens"></A>
 <br>
 <div class=routine>
@@ -406,17 +910,21 @@
 <!-- Description -->
 
 <P>
-Copies the contents of ens1 into ens2. If the num_copies and num_vars are
-not consistent or if the distribution_type is not consistent, fails with an
-error. If duplicate_time is true, the times from ens1 are copied over the
-times of ens2.
+Copies the contents of the vars array from ens1 into ens2. 
+If the num_copies and num_vars are not consistent or if the distribution_type 
+is not consistent, fails with an error. 
+If duplicate_time is true, the times from ens1 are copied over the
+times of ens2.  Only the vars array data is copied from the source to the
+destination.  Transpose the data after duplication if you want to access
+the copies.
 </P>
 
-<TABLE border=0 cellpadding=3 width=100%>
+<TABLE border=0 cellpadding=3 width=100% summary=''>
 <TR><TD valign=top><em class=code>ens1</em></TD>
-    <TD>Ensemble handle of ensemble to be copies into ens2.</TD></TR>
+    <TD>Ensemble handle of ensemble to be copies into ens2.
+        Data from the vars array will be replicated.</TD></TR>
 <TR><TD valign=top><em class=code>ens2</em></TD>
-    <TD>Ensemble handle of ensemble into which ens1 is copied.</TD></TR>
+    <TD>Ensemble handle of ensemble into which ens1 vars data will be copied.</TD></TR>
 <TR><TD valign=top><em class=code>duplicate_time&nbsp;&nbsp;&nbsp;</em></TD>
     <TD>If true, copy the times from ens1 into ens2, else leave ens2 times unchanged.</TD></TR>
 </TABLE>
@@ -431,8 +939,9 @@
 <div class=routine>
 <em class=call> var = get_my_num_copies(ens_handle) </em>
 <pre>
-integer, intent(out)            :: <em class=code>get_my_num_copies</em>
-type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+integer                          :: <em class=code>get_my_num_copies</em>
+type(ensemble_type), intent(in)  :: <em class=code>ens_handle</em>
+
 </pre>
 </div>
 
@@ -441,7 +950,8 @@
 
 <P>
 Returns number of copies stored by this process when storing all
-variables for a subset of copies.
+variables for a subset of copies.  Same as num_copies if running
+with only a single process.
 </P>
 
 <TABLE width=100% border=0 summary="" cellpadding=3>
@@ -462,7 +972,7 @@
 <div class=routine>
 <em class=call> var = get_my_num_vars(ens_handle) </em>
 <pre>
-integer, intent(out)            :: <em class=code>get_my_num_vars</em>
+integer                         :: <em class=code>get_my_num_vars</em>
 type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
 </pre>
 </div>
@@ -472,7 +982,8 @@
 
 <P>
 Returns number of variables stored by this process when storing all
-copies of a subset of variables.
+copies of a subset of variables.  Same as num_vars if running
+with only a single process.
 </P>
 
 <TABLE width=100% border=0 summary="" cellpadding=3>
@@ -494,7 +1005,7 @@
 <em class=call> call get_my_copies(ens_handle, copies) </em>
 <pre>
 type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
-integer, intent(out)            :: <em class=code>copies</em>
+integer, intent(out)            :: <em class=code>copies(:)</em>
 </pre>
 </div>
 
@@ -502,8 +1013,8 @@
 <!-- Description -->
 
 <P>
-Returns a list of all copies stored on this process when storing subset of copies
-of all variables.
+Returns a list of the global copy numbers stored on this process 
+when storing subset of copies of all variables.
 </P>
 
 <TABLE width=100% border=0 summary="" cellpadding=3>
@@ -522,10 +1033,10 @@
 <A NAME="get_my_vars"></A>
 <br>
 <div class=routine>
-<em class=call> call get_my_vars(ens_handle, copies) </em>
+<em class=call> call get_my_vars(ens_handle, vars) </em>
 <pre>
 type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
-integer, intent(out)            :: <em class=code>vars</em>
+integer, intent(out)            :: <em class=code>vars(:)</em>
 </pre>
 </div>
 
@@ -533,8 +1044,8 @@
 <!-- Description -->
 
 <P>
-Returns a list of all variables stored on this process when storing all copies
-of a subset of variables.
+Returns a list of the global variable numbers stored on this process 
+when storing all copies of a subset of variables.
 </P>
 
 <TABLE width=100% border=0 summary="" cellpadding=3>
@@ -550,15 +1061,184 @@
 
 <!--===================== DESCRIPTION OF A ROUTINE =====================-->
 
+<A NAME="get_copy_owner_index"></A>
+<br>
+<div class=routine>
+<em class=call> call get_copy_owner_index(copy_number, owner, owners_index) </em>
+<pre>
+integer, intent(in)  :: <em class=code>copy_number</em>
+integer, intent(out) :: <em class=code>owner</em>
+integer, intent(out) :: <em class=code>owners_index</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Given the global index of a copy number, returns the PE that
+stores this copy when all variables of a subset of copies are stored and
+the local storage index for this copy on that process.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>copy_number</em></TD>
+    <TD>Global index of a copy from an ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>owner</em></TD>
+    <TD>Process Element (PE) that stores this copy when each has all 
+         variables of a subset of copies.</TD></TR>
+<TR><TD valign=top><em class=code>owners_index&nbsp;&nbsp;&nbsp;</em></TD>
+    <TD>Local storage index for this copy on the owning process.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="get_var_owner_index"></A>
+<br>
+<div class=routine>
+<em class=call> call get_var_owner_index(var_number, owner, owners_index) </em>
+<pre>
+integer, intent(in)  :: <em class=code>var_number</em>
+integer, intent(out) :: <em class=code>owner</em>
+integer, intent(out) :: <em class=code>owners_index</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Given the global index of a variable in the vector, returns the PE that
+stores this variable when all copies of a subset of variables are stored and
+the local storage index for this variable on that process.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>var_number</em></TD>
+    <TD>Global index of a variable in the vector from an ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>owner</em></TD>
+    <TD>Process Element (PE) that stores this variable when each has all 
+         copies of subset of variables.</TD></TR>
+<TR><TD valign=top><em class=code>owners_index&nbsp;&nbsp;&nbsp;</em></TD>
+    <TD>Local storage index for this variable on the owning process.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="all_vars_to_all_copies"></A>
+<br>
+<div class=routine>
+<em class=call> call all_vars_to_all_copies(ens_handle, label) </em>
+<pre>
+type(ensemble_type), intent(inout)        :: <em class=code>ens_handle</em>
+character(len=*),    intent(in), optional :: <em class=code>label</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Transposes data from a representation in which each PE has a subset of
+copies of all variables to one in which each has all copies of a subset of
+variables. In the current implementation, storage is not released so both
+representations are always available. However, one representation may be
+current while the other is out of date.
+</P>
+<P>
+Different different numbers of copies, different lengths of the vectors,
+different numbers of PEs and different implementations of the MPI parallel

@@ Diff output truncated at 40000 characters. @@


More information about the Dart-dev mailing list