[Dart-dev] [6495] DART/trunk/ensemble_manager/ensemble_manager_mod.html: major updating to make the docs consistent with the
nancy at ucar.edu
nancy at ucar.edu
Wed Sep 25 16:16:34 MDT 2013
Revision: 6495
Author: nancy
Date: 2013-09-25 16:16:33 -0600 (Wed, 25 Sep 2013)
Log Message:
-----------
major updating to make the docs consistent with the
current state of the ensemble manager code.
Modified Paths:
--------------
DART/trunk/ensemble_manager/ensemble_manager_mod.html
-------------- next part --------------
Modified: DART/trunk/ensemble_manager/ensemble_manager_mod.html
===================================================================
--- DART/trunk/ensemble_manager/ensemble_manager_mod.html 2013-09-25 21:34:23 UTC (rev 6494)
+++ DART/trunk/ensemble_manager/ensemble_manager_mod.html 2013-09-25 22:16:33 UTC (rev 6495)
@@ -52,6 +52,34 @@
and quality control and ensembles of forward observation operator error
status.
</P>
+<P>
+The ensemble manager interacts strongly with the multiple process capability
+of the Message Passing Interface (MPI) libraries. It is used to partition
+the data so each MPI process stores only a subset of the copies and variables,
+dividing the data as evenly as possible across the processes. At no time
+during the execution does any one process have to store the entire dataset
+for all ensemble members (unless running in serial mode without MPI, or if
+running with 1 MPI task).
+</P>
+<P>
+The ensemble manager is set of general purpose data management routines.
+For run-time efficiency, the derived type information is not marked private which
+means other modules can directly manipulate the data arrays.
+However it means much care must be taken to access the most recently
+updated representation of the data, either the copies or variables arrays.
+</P>
+<P>
+A set of sanity check routines have been added to track the last modified
+version of the data: the copies array or the vars array. Before directly
+reading or writing these arrays call one of the 'prepare' routines to
+indicate what kind of data access you are about to make. If the most recently
+updated data is not as expected an error message will occur. After the
+direct access if the following operations detect that the data they are
+operating on is not the most recently updated they will print an error message.
+Routines inside the ensemble manager that alter the copies or vars will set
+the state automatically so these routines are only necessary to call if you
+are directly accessing the copies or vars arrays from outside the ensemble manager.
+</P>
<!--==================================================================-->
<!--=================== DESCRIPTION OF A NAMELIST ===================-->
@@ -133,7 +161,7 @@
<TR><TD>layout</TD>
<TD>integer</TD>
<TD>Determines the process (pe) layout on MPI tasks.
- 1 is pe = MPI task. 2 is a round-robin layout around the nodes. Layout 2
+ 1 is PE = MPI task. 2 is a round-robin layout around the nodes. Layout 2
results in a more even usage of memory across nodes. This may allow you to run
with a larger state vector without hitting the memory limit of the node.
It may give a slight (5%) increase in performance, but this is machine dependent.
@@ -190,8 +218,10 @@
<div>
<table width="100%" summary='communication patterns'><tr>
-<td><a href="../doc/images/comm_pattern96.png"><img src="../doc/images/comm_pattern96.png" width="400"></a>
-<td><a href="../doc/images/comm_pattern512.png"><img src="../doc/images/comm_pattern512.png" width="400"></a>
+<td><a href="../doc/images/comm_pattern96.png">
+<img src="../doc/images/comm_pattern96.png" width="400" alt="communication pattern"></a>
+<td><a href="../doc/images/comm_pattern512.png">
+<img src="../doc/images/comm_pattern512.png" width="400" alt="communication pattern 2"></a>
</table>
</div>
@@ -209,6 +239,7 @@
time_manager_mod
random_seq_mod
mpi_utilities_mod
+sort_mod
</PRE>
<!--==================================================================-->
@@ -222,27 +253,38 @@
<div class="top">[<a href="#">top</a>]</div><hr />
<H2>PUBLIC INTERFACES</H2>
-<TABLE>
+<TABLE summary='public routine list'>
<TR><TD><em class=call>use ensemble_manager_mod, only : </em></TD>
- <TD><A HREF="#all_copies_to_all_vars">all_copies_to_all_vars</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#all_vars_to_all_copies">all_vars_to_all_copies</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#compute_copy_mean">compute_copy_mean</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#compute_copy_mean_sd">compute_copy_mean_sd</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#compute_copy_mean_var">compute_copy_mean_var</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#duplicate_ens">duplicate_ens</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#end_ensemble_manager">end_ensemble_manager</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#ensemble_type">ensemble_type</A></TD></TR>
+ <TD><A HREF="#init_ensemble_manager">init_ensemble_manager</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#read_ensemble_restart">read_ensemble_restart</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#write_ensemble_restart">write_ensemble_restart</A></TD></TR>
<TR><TD> </TD><TD><A HREF="#get_copy">get_copy</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#put_copy">put_copy</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#broadcast_copy">broadcast_copy</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#set_ensemble_time">set_ensemble_time</A></TD></TR>
<TR><TD> </TD><TD><A HREF="#get_ensemble_time">get_ensemble_time</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#end_ensemble_manager">end_ensemble_manager</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#duplicate_ens">duplicate_ens</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#get_my_num_copies">get_my_num_copies</A></TD></TR>
<TR><TD> </TD><TD><A HREF="#get_my_copies">get_my_copies</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#get_my_num_copies">get_my_num_copies</A></TD></TR>
<TR><TD> </TD><TD><A HREF="#get_my_num_vars">get_my_num_vars</A></TD></TR>
<TR><TD> </TD><TD><A HREF="#get_my_vars">get_my_vars</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#get_copy_owner_index">get_copy_owner_index</A></TD></TR>
<TR><TD> </TD><TD><A HREF="#get_var_owner_index">get_var_owner_index</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#init_ensemble_manager">init_ensemble_manager</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#put_copy">put_copy</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#read_ensemble_restart">read_ensemble_restart</A></TD></TR>
-<TR><TD> </TD><TD><A HREF="#write_ensemble_restart">write_ensemble_restart</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#all_vars_to_all_copies">all_vars_to_all_copies</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#all_copies_to_all_vars">all_copies_to_all_vars</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#compute_copy_mean">compute_copy_mean</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#compute_copy_mean_sd">compute_copy_mean_sd</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#compute_copy_mean_var">compute_copy_mean_var</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#prepare_to_write_to_vars">prepare_to_write_to_vars</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#prepare_to_write_to_copies">prepare_to_write_to_copies</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#prepare_to_read_from_vars">prepare_to_read_from_vars</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#prepare_to_read_from_copies">prepare_to_read_from_copies</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#prepare_to_update_vars">prepare_to_update_vars</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#prepare_to_update_copies">prepare_to_update_copies</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#print_ens_handle">print_ens_handle</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#map_pe_to_task">map_pe_to_task</A></TD></TR>
+<TR><TD> </TD><TD><A HREF="#map_task_to_pe">map_task_to_pe</A></TD></TR>
</TABLE>
<P>
@@ -253,23 +295,32 @@
<!--=================== DESCRIPTION OF A LOCAL TYPE ==================-->
-<!--kdr no private in code -->
<A NAME="ensemble_type"></A>
<BR>
<div class=type>
<pre>
<em class=call>type ensemble_type</em>
- !!! private
+ !DIRECT ACCESS INTO STORAGE IS ALLOWED; BE CAREFUL
integer :: num_copies
integer :: num_vars
integer :: my_num_copies
integer :: my_num_vars
integer, pointer :: my_copies(:)
integer, pointer :: my_vars(:)
- real(r8), pointer :: copies(:, :)
- real(r8), pointer :: vars(:, :)
+ ! Storage in next line is to be used when each PE has all copies of subset of vars
+ real(r8), pointer :: copies(:, :) ! Dimensioned (num_copies, my_num_vars)
+ ! Storage on next line is used when each PE has subset of copies of all vars
+ real(r8), pointer :: vars(:, :) ! Dimensioned (num_vars, my_num_copies)
+ ! Time is only related to var complete
type(time_type), pointer :: time(:)
integer :: distribution_type
+ integer :: valid ! copies modified last, vars modified last, both same
+ integer :: id_num
+ integer, allocatable :: task_to_pe_list(:) ! List of tasks
+ integer, allocatable :: pe_to_task_list(:) ! List of tasks
+ ! Flexible my_pe, layout_type which allows different task layouts for different ensemble handles
+ integer :: my_pe
+ integer :: layout_type
end type ensemble_type
</pre>
</div>
@@ -283,35 +334,69 @@
used throughout DART.
</P>
-<TABLE border=0 cellpadding=3 width=100%>
-<TR><TH align=left>Component </TH>
- <TH align=left>Description </TH></TR>
-<TR><TD valign=top> num_copies </TD>
+<TABLE border=0 cellpadding=3 width=100% summary='derived type description'>
+<THEAD align=left>
+<TR><TH> Component </TH>
+ <TH> Description </TH> </TR>
+</THEAD>
+
+<TBODY valign=top>
+
+<TR><TD> num_copies </TD>
<TD> Global number of copies of the vector. </TD></TR>
-<TR><TD valign=top> num_vars </TD>
+<TR><TD> num_vars </TD>
<TD> Global number of elements (variables) in the vector. </TD></TR>
-<TR><TD valign=top> my_num_copies </TD>
+<TR><TD> my_num_copies </TD>
<TD> Number of copies stored by this process. </TD></TR>
-<TR><TD valign=top> my_num_vars </TD>
+<TR><TD> my_num_vars </TD>
<TD> Number of variables stored by this process. </TD></TR>
-<TR><TD valign=top> my_copies </TD>
+<TR><TD> my_copies </TD>
<TD> Dimensioned to size my_num_copies. Contains a list of the global
indices of copies stored by this process. </TD></TR>
-<TR><TD valign=top> my_vars </TD>
+<TR><TD> my_vars </TD>
<TD> Dimensioned to size my_num_vars. Contains a list of the global
indices of variables stored by this process. </TD></TR>
-<TR><TD valign=top> copies </TD>
+<TR><TD> copies </TD>
<TD> Dimensioned (num_copies, my_num_vars). Storage for all copies
of variables stored by this process. </TD></TR>
-<TR><TD valign=top> vars </TD>
+<TR><TD> vars </TD>
<TD> Dimensioned (num_vars, my_num_copies). Storage for all variables
of copies stored by this process. </TD></TR>
-<TR><TD valign=top> time </TD>
+<TR><TD> time </TD>
<TD> Dimensioned my_num_copies. A time_type that stores time associated
with a given copy of the vector. </TD></TR>
-<TR><TD valign=top> distribution_type </TD>
+<TR><TD> distribution_type </TD>
<TD>Does nothing at present. Can be used for future releases to control
the layout of different copies and variables in storage. </TD></TR>
+<TR><TD> valid </TD>
+ <TD> Flag to track whether the copies array has the most recently updated
+ data, the vars array is most recently modified, or if both the arrays
+ have identical data, like after a transpose. </TD></TR>
+<TR><TD> id_num </TD>
+ <TD> Internal number unique to each ensemble handle, used for
+ debugging purposes. </TD></TR>
+<TR><TD> task_to_pe_list </TD>
+ <TD> Mapping from MPI task number to logical Processing Element (PE) number.
+ Enables different assignment of MPI tasks to PEs. If the number of MPI
+ tasks is larger than the number of copies of the vector, when the ensemble
+ is var complete then the first N MPI tasks have allocated 'vars' arrays
+ and the remaining ones do not. Assigning the MPI tasks round-robin
+ to multi-processor nodes can make the memory usage more uniform across
+ nodes, which may allow more MPI tasks per node than the standard layout. </TD></TR>
+<TR><TD> pe_to_task_list </TD>
+ <TD> Logical PE to MPI task mapping. See above for more description.
+ </TD></TR>
+<TR><TD> my_pe </TD>
+ <TD> The logical PE number for the MPI task.
+ </TD></TR>
+<TR><TD> layout_type </TD>
+ <TD> Controls the mapping type between MPI tasks and PEs. Currently type 1
+ is the standard layout (one-to-one mapping) and type 2 is a round-robin
+ mapping where each node gets a task in turn before assigning a second
+ task to each node, until all tasks are assigned.
+ </TD></TR>
+
+</TBODY>
</TABLE>
</div>
@@ -322,13 +407,15 @@
<A NAME="init_ensemble_manager"></A>
<br>
<div class=routine>
-<em class=call> call init_ensemble_manager(ens_handle, num_copies,
- num_vars <em class=optionalcode>[, distribution_type_in]</em>) </em>
+<em class=call> call init_ensemble_manager(ens_handle, num_copies, num_vars
+<em class=optionalcode>[, distribution_type_in]</em>
+<em class=optionalcode>[, layout_type]</em>) </em>
<pre>
type(ensemble_type), intent(out) :: <em class=code>ens_handle</em>
-integer, intent(in) :: <em class=code>num_copies</em>
-integer, intent(in) :: <em class=code>num_vars</em>
-integer, optional, intent(in) :: <em class=optionalcode>distribution_type_in</em>
+integer, intent(in) :: <em class=code>num_copies</em>
+integer, intent(in) :: <em class=code>num_vars</em>
+integer, optional, intent(in) :: <em class=optionalcode>distribution_type_in</em>
+integer, optional, intent(in) :: <em class=optionalcode>layout_type</em>
</pre>
</div>
@@ -347,8 +434,14 @@
<TD>Number of copies of vector.</TD></TR>
<TR><TD valign=top><em class=code>num_vars</em></TD>
<TD>Number of variables in the vector.</TD></TR>
-<TR><TD valign=top><em class=optionalcode>distribution_type_in </em></TD>
- <TD>Controls layout of storage on pe's. Currently only option 1 is supported.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>distribution_type_in</em></TD>
+ <TD>Controls layout of storage on PEs. Currently only option 1 is supported.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>layout_type</em></TD>
+ <TD>Controls layout of MPI tasks on PEs. Type 1 is the default, where MPI tasks
+ are assigned to PEs on a one-to-one basis. Type 2 is a round-robin assignment
+ where each node gets one task before the nodes are assigned a second task.
+ If running with more MPI tasks than <em class=code>num_copies</em>, this can
+ result in a more uniform usage of memory across the nodes. </TD></TR>
</TABLE>
</div>
@@ -356,14 +449,21 @@
<!--===================== DESCRIPTION OF A ROUTINE =====================-->
-<A NAME="get_var_owner_index"></A>
+<A NAME="read_ensemble_restart"></A>
<br>
<div class=routine>
-<em class=call> call get_var_owner_index(var_number, owner, owners_index) </em>
+<em class=call> call read_ensemble_restart(ens_handle, start_copy, end_copy,
+ start_from_restart, file_name
+<em class=optionalcode>[, init_time]</em>
+<em class=optionalcode>[, force_single_file]</em>) </em>
<pre>
-integer, intent(in) :: <em class=code>var_number</em>
-integer, intent(out) :: <em class=code>owner</em>
-integer, intent(out) :: <em class=code>owners_index</em>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+integer, intent(in) :: <em class=code>start_copy</em>
+integer, intent(in) :: <em class=code>end_copy</em>
+logical, intent(in) :: <em class=code>start_from_restart</em>
+character(len=*), intent(in) :: <em class=code>file_name</em>
+type(time_type), optional, intent(in) :: <em class=optionalcode>init_time</em>
+logical, optional, intent(in) :: <em class=optionalcode>force_single_file</em>
</pre>
</div>
@@ -371,19 +471,41 @@
<!-- Description -->
<P>
-Given the global index of a variable in the vector, returns the process that
-stores this variable when all copies of a subset of variables are stored and
-the local storage index for this variable on that process.
+Read in a set of copies of a vector from file file_name. The copies read are
+place into global copies start_copy:end_copy in the ens_handle.
+If start_from_restart is false, then only a single copy of the vector is
+read from the file and then it is perturbed using routines in assim_model_mod
+to generate the required number of copies. The read can be from a single file
+that contains all needed copies or from a different file for each copy. This
+choice is controlled by the namelist entry single_restart_file_in. However, the
+optional argument force_single_file forces the read to be from a single file
+if it is present and true. This is used for ensembles that contain the
+inflation values for state space inflation. If multiple files are to be read,
+the file names are generated by appending integers to the input file_name.
+If the input is a single file all reads are done sequentially by process 0
+and then shipped to the PE that stores that copy.
+If the input is multiple files each MPI task reads the copies it stores
+directly and independently.
</P>
<TABLE width=100% border=0 summary="" cellpadding=3>
-<TR><TD valign=top><em class=code>var_number</em></TD>
- <TD>Global index of a variable in the vector from an ensemble.</TD></TR>
-<TR><TD valign=top><em class=code>owner</em></TD>
- <TD>Process (0 to num_processes) that stores this variable when each has all
- copies of subset of variables.</TD></TR>
-<TR><TD valign=top><em class=code>owners_index </em></TD>
- <TD>Local storage index for this variable on the owning process.</TD></TR>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle of ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>start_copy </em></TD>
+ <TD>Global index of first of continguous set of copies to be read.</TD></TR>
+<TR><TD valign=top><em class=code>end_copy </em></TD>
+ <TD>Global index of last of contiguous set of copies to be read,
+ copies(start_copy:end_copy).</TD></TR>
+<TR><TD valign=top><em class=code>start_from_restart </em></TD>
+ <TD>If true, read all copies from file. If false, read one copy and perturb
+ to get required number.</TD></TR>
+<TR><TD valign=top><em class=code>file_name </em></TD>
+ <TD>Name of file from which to read.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>init_time </em></TD>
+ <TD>If present, set time of all copies read to this value.</TD></TR>
+<TR><TD valign=top><em class=optionalcode>force_single_file </em></TD>
+ <TD>If present and true, force the read to be from a single file which
+ contains all copies.</TD></TR>
</TABLE>
</div>
@@ -391,6 +513,388 @@
<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+<A NAME="write_ensemble_restart"></A>
+<br>
+<div class=routine>
+<em class=call> call write_ensemble_restart(ens_handle, file_name, start_copy,
+ end_copy <em class=optionalcode>[, force_single_file]</em>) </em>
+<pre>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+character(len=*), intent(in) :: <em class=code>file_name</em>
+integer, intent(in) :: <em class=code>start_copy</em>
+integer, intent(in) :: <em class=code>end_copy</em>
+logical, optional, intent(in) :: <em class=optionalcode>force_single_file</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Writes a set of copies of a vector to file file_name. The copies written are
+from global copies start_copy:end_copy in the ens_handle.
+The write can be to a single file or to a different file for each copy. This
+choice is controlled by the namelist entry single_restart_file_out. However, the
+optional argument force_single_file forces the write to be to a single file
+if it is present and true. This is used for ensembles that contain the
+inflation values for state space inflation. If multiple files are to be written,
+the file names are generated by appending integers to the input file_name.
+If the output is a single file all copies are shipped from the PE that
+stores that copy to process 0, and then written out sequentially.
+If the output is to multiple files each MPI task writes the copies it stores
+directly and independently.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle of ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>file_name </em></TD>
+ <TD>Name of file from which to read.</TD></TR>
+<TR><TD valign=top><em class=code>start_copy </em></TD>
+ <TD>Global index of first of continguous set of copies to be written.</TD></TR>
+<TR><TD valign=top><em class=code>end_copy </em></TD>
+ <TD>Global index of last of contiguous set of copies to be written,
+ copies(start_copy:end_copy).</TD></TR>
+<TR><TD valign=top><em class=optionalcode>force_single_file </em></TD>
+ <TD>If present and true, force the write to be to a single file which
+ contains all copies.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="get_copy"></A>
+<br>
+<div class=routine>
+<em class=call> call get_copy(receiving_pe, ens_handle, copy, vars
+<em class=optionalcode>[, mtime]</em>) </em>
+<pre>
+integer, intent(in) :: <em class=code>receiving_pe</em>
+type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+integer, intent(in) :: <em class=code>copy</em>
+real(r8), dimension(:), intent(out) :: <em class=code>vars</em>
+type(time_type), optional, intent(out) :: <em class=optionalcode>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Retrieves a copy of the state vector, indexed by the global index copy. The
+process that is to receive the copy is receiving_pe and the copy is returned
+in the one dimensional array vars. The time of the copy is also returned if
+mtime is present. This is generally used for operations, like IO, that require
+a single processor to do things with the entire state vector.
+Data is only returned in vars on the receiving PE; vars on all other PEs is unset.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>receiving_pe </em></TD>
+ <TD>This process ends up with the requested copy of the state
+ vector.</TD></TR>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy </em></TD>
+ <TD>The global index of the copy of the state vector that is to be
+ retrieved.</TD></TR>
+<TR><TD valign=top><em class=code>vars </em></TD>
+ <TD>One dimensional array in which the requested copy of the state vector
+ is returned. Data is only returned in vars on the receiving PE;
+ vars on all other PEs is unset. </TD></TR>
+<TR><TD valign=top><em class=optionalcode>mtime </em></TD>
+ <TD>If present returns the time of the requested copy.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="put_copy"></A>
+<br>
+<div class=routine>
+<em class=call> call put_copy(sending_pe, ens_handle, copy, vars
+<em class=optionalcode>[, mtime]</em>) </em>
+<pre>
+integer, intent(in) :: <em class=code>sending_pe</em>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+integer, intent(in) :: <em class=code>copy</em>
+real(r8), dimension(:), intent(in) :: <em class=code>vars</em>
+type(time_type), optional, intent(in) :: <em class=optionalcode>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Sends a state vector, in vars, from the given process to the process
+storing the global index copy.
+The time of the copy is also sent if mtime is present.
+This is generally used for operations, like IO, that require
+a single processor to do things with the entire state vector. For instance,
+if a single process reads in a state vector, it can be shipped to the storing
+process by this subroutine. Only the data in vars on the sending PE
+is processed; vars on all other PEs is ignored.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>sending_pe </em></TD>
+ <TD>This process sends the copy of the state vector.</TD></TR>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy </em></TD>
+ <TD>The global index of the copy of the state vector that is to be sent.
+</TD></TR>
+<TR><TD valign=top><em class=code>vars </em></TD>
+ <TD>One dimensional array in which the requested copy of the state vector
+ is located. Only the data in vars on the sending PE
+ is processed; vars on all other PEs is ignored. </TD></TR>
+<TR><TD valign=top><em class=optionalcode>mtime </em></TD>
+ <TD>If present send the time of the copy.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="broadcast_copy"></A>
+<br>
+<div class=routine>
+<em class=call> call broadcast_copy(ens_handle, copy, arraydata)</em>
+<pre>
+type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+integer, intent(in) :: <em class=code>copy</em>
+real(r8), dimension(:), intent(out) :: <em class=code>arraydata</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Finds which PE has the global index copy and broadcasts
+that copy to all PEs. <em class=code>arraydata</em> is an output on
+all PEs, even on the PE which is the owner if it is separate
+storage from the vars array in the ensemble handle.
+This is a collective routine, which means it must be called by
+all processes in the job.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy </em></TD>
+ <TD>The global index of the copy of the state vector that is to be sent.
+</TD></TR>
+<TR><TD valign=top><em class=code>arraydata </em></TD>
+ <TD>One dimensional array into which the requested copy of the state vector
+ will be copied on all PEs, including the sending PE.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="set_ensemble_time"></A>
+<br>
+<div class=routine>
+<em class=call> call set_ensemble_time(ens_handle, indx, mtime) </em>
+<pre>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+integer, intent(in) :: <em class=code>indx</em>
+type(time_type), intent(in) :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Set the time of a copy to the given value. <em class=code>indx</em> in
+this case is the local copy number for a specific task.
+<!-- Contrast this with
+<a href="#set_copy_time">set_copy_time()</a> which uses global copy numbers. -->
+<a href="#get_copy_owner_index">get_copy_owner_index()</a>
+can be called to see if you are the owning task for a given global copy number,
+and to get the local index number for that copy.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>indx </em></TD>
+ <TD>The local index of the copy of the state vector that is to be set.
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime </em></TD>
+ <TD>The time to set for this copy. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="get_ensemble_time"></A>
+<br>
+<div class=routine>
+<em class=call> call get_ensemble_time(ens_handle, indx, mtime) </em>
+<pre>
+type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+integer, intent(in) :: <em class=code>indx</em>
+type(time_type), intent(out) :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Get the time associated with a copy. <em class=code>indx</em> in
+this case is the local copy number for a specific task.
+<a href="#get_copy_owner_index">get_copy_owner_index()</a>
+can be called to see if you are the owning task for a given global copy number,
+and to get the local index number for that copy.
+<!-- Contrast this with
+<a href="#get_copy_time">get_copy_time()</a> which uses global copy numbers. -->
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>indx </em></TD>
+ <TD>The local index of the copy to retrieve the time from.
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime </em></TD>
+ <TD>The returned time value. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--
+
+this is probably a good idea, but the code hasn't been tested yet
+and it's close to a release, so leave it out for now.
+
+....===================== DESCRIPTION OF A ROUTINE =====================...
+
+<A NAME="set_copy_time"></A>
+<br>
+<div class=routine>
+<em class=call> call set_copy_time(ens_handle, copy, mtime) </em>
+<pre>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+
+type(time_type), intent(in) :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+< ! - - Description - - >
+
+<P>
+Set the time of a copy to the given value. <em class=code>copy</em> in
+this case is the global copy number. Contrast this with
+<a href="#set_ensemble_time">set_ensemble_time()</a> which uses local
+copy indices.
+This routine must be called by all processes and with the same mtime
+but only the process that stores the requested copy will do the update.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy </em></TD>
+ <TD>The global number of the copy that is to be set.
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime </em></TD>
+ <TD>The time to set for this copy. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+....===================== DESCRIPTION OF A ROUTINE =====================...
+
+not sure if this is such a good idea. only the task that actually
+stores the copy will return it, unless we do some communication.
+should we make it communicate?
+
+<A NAME="get_copy_time"></A>
+<br>
+<div class=routine>
+<em class=call> call get_copy_time(ens_handle, copy, mtime) </em>
+<pre>
+type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+integer, intent(in) :: <em class=code>copy</em>
+type(time_type), intent(out) :: <em class=code>mtime</em>
+</pre>
+</div>
+
+<div class=indent1>
+< ! - - Description - - >
+
+<P>
+Get the time associated with a copy. <em class=code>copy</em> in
+this case is the global copy number. Contrast this with
+<a href="#get_ensemble_time">get_ensemble_time()</a> which uses local
+copy indices.
+This routine can be called by all processes and only the
+process that stores the requested copy will return a value so it
+must be combined with a call to get_copy_owner_index() to see if
+you are the owner.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>copy </em></TD>
+ <TD>The global index of the copy to retrieve the time from.
+</TD></TR>
+<TR><TD valign=top><em class=code>mtime </em></TD>
+ <TD>The returned time value. </TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+-->
+
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="end_ensemble_manager"></A>
+<br>
+<div class=routine>
+<em class=call> call end_ensemble_manager(ens_handle) </em>
+<pre>
+type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Frees up storage associated with an ensemble.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>ens_handle </em></TD>
+ <TD>Handle for an ensemble.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
<A NAME="duplicate_ens"></A>
<br>
<div class=routine>
@@ -406,17 +910,21 @@
<!-- Description -->
<P>
-Copies the contents of ens1 into ens2. If the num_copies and num_vars are
-not consistent or if the distribution_type is not consistent, fails with an
-error. If duplicate_time is true, the times from ens1 are copied over the
-times of ens2.
+Copies the contents of the vars array from ens1 into ens2.
+If the num_copies and num_vars are not consistent or if the distribution_type
+is not consistent, fails with an error.
+If duplicate_time is true, the times from ens1 are copied over the
+times of ens2. Only the vars array data is copied from the source to the
+destination. Transpose the data after duplication if you want to access
+the copies.
</P>
-<TABLE border=0 cellpadding=3 width=100%>
+<TABLE border=0 cellpadding=3 width=100% summary=''>
<TR><TD valign=top><em class=code>ens1</em></TD>
- <TD>Ensemble handle of ensemble to be copies into ens2.</TD></TR>
+ <TD>Ensemble handle of ensemble to be copies into ens2.
+ Data from the vars array will be replicated.</TD></TR>
<TR><TD valign=top><em class=code>ens2</em></TD>
- <TD>Ensemble handle of ensemble into which ens1 is copied.</TD></TR>
+ <TD>Ensemble handle of ensemble into which ens1 vars data will be copied.</TD></TR>
<TR><TD valign=top><em class=code>duplicate_time </em></TD>
<TD>If true, copy the times from ens1 into ens2, else leave ens2 times unchanged.</TD></TR>
</TABLE>
@@ -431,8 +939,9 @@
<div class=routine>
<em class=call> var = get_my_num_copies(ens_handle) </em>
<pre>
-integer, intent(out) :: <em class=code>get_my_num_copies</em>
-type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+integer :: <em class=code>get_my_num_copies</em>
+type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
+
</pre>
</div>
@@ -441,7 +950,8 @@
<P>
Returns number of copies stored by this process when storing all
-variables for a subset of copies.
+variables for a subset of copies. Same as num_copies if running
+with only a single process.
</P>
<TABLE width=100% border=0 summary="" cellpadding=3>
@@ -462,7 +972,7 @@
<div class=routine>
<em class=call> var = get_my_num_vars(ens_handle) </em>
<pre>
-integer, intent(out) :: <em class=code>get_my_num_vars</em>
+integer :: <em class=code>get_my_num_vars</em>
type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
</pre>
</div>
@@ -472,7 +982,8 @@
<P>
Returns number of variables stored by this process when storing all
-copies of a subset of variables.
+copies of a subset of variables. Same as num_vars if running
+with only a single process.
</P>
<TABLE width=100% border=0 summary="" cellpadding=3>
@@ -494,7 +1005,7 @@
<em class=call> call get_my_copies(ens_handle, copies) </em>
<pre>
type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
-integer, intent(out) :: <em class=code>copies</em>
+integer, intent(out) :: <em class=code>copies(:)</em>
</pre>
</div>
@@ -502,8 +1013,8 @@
<!-- Description -->
<P>
-Returns a list of all copies stored on this process when storing subset of copies
-of all variables.
+Returns a list of the global copy numbers stored on this process
+when storing subset of copies of all variables.
</P>
<TABLE width=100% border=0 summary="" cellpadding=3>
@@ -522,10 +1033,10 @@
<A NAME="get_my_vars"></A>
<br>
<div class=routine>
-<em class=call> call get_my_vars(ens_handle, copies) </em>
+<em class=call> call get_my_vars(ens_handle, vars) </em>
<pre>
type(ensemble_type), intent(in) :: <em class=code>ens_handle</em>
-integer, intent(out) :: <em class=code>vars</em>
+integer, intent(out) :: <em class=code>vars(:)</em>
</pre>
</div>
@@ -533,8 +1044,8 @@
<!-- Description -->
<P>
-Returns a list of all variables stored on this process when storing all copies
-of a subset of variables.
+Returns a list of the global variable numbers stored on this process
+when storing all copies of a subset of variables.
</P>
<TABLE width=100% border=0 summary="" cellpadding=3>
@@ -550,15 +1061,184 @@
<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+<A NAME="get_copy_owner_index"></A>
+<br>
+<div class=routine>
+<em class=call> call get_copy_owner_index(copy_number, owner, owners_index) </em>
+<pre>
+integer, intent(in) :: <em class=code>copy_number</em>
+integer, intent(out) :: <em class=code>owner</em>
+integer, intent(out) :: <em class=code>owners_index</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Given the global index of a copy number, returns the PE that
+stores this copy when all variables of a subset of copies are stored and
+the local storage index for this copy on that process.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>copy_number</em></TD>
+ <TD>Global index of a copy from an ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>owner</em></TD>
+ <TD>Process Element (PE) that stores this copy when each has all
+ variables of a subset of copies.</TD></TR>
+<TR><TD valign=top><em class=code>owners_index </em></TD>
+ <TD>Local storage index for this copy on the owning process.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="get_var_owner_index"></A>
+<br>
+<div class=routine>
+<em class=call> call get_var_owner_index(var_number, owner, owners_index) </em>
+<pre>
+integer, intent(in) :: <em class=code>var_number</em>
+integer, intent(out) :: <em class=code>owner</em>
+integer, intent(out) :: <em class=code>owners_index</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Given the global index of a variable in the vector, returns the PE that
+stores this variable when all copies of a subset of variables are stored and
+the local storage index for this variable on that process.
+</P>
+
+<TABLE width=100% border=0 summary="" cellpadding=3>
+<TR><TD valign=top><em class=code>var_number</em></TD>
+ <TD>Global index of a variable in the vector from an ensemble.</TD></TR>
+<TR><TD valign=top><em class=code>owner</em></TD>
+ <TD>Process Element (PE) that stores this variable when each has all
+ copies of subset of variables.</TD></TR>
+<TR><TD valign=top><em class=code>owners_index </em></TD>
+ <TD>Local storage index for this variable on the owning process.</TD></TR>
+</TABLE>
+
+</div>
+<br>
+
+<!--===================== DESCRIPTION OF A ROUTINE =====================-->
+
+<A NAME="all_vars_to_all_copies"></A>
+<br>
+<div class=routine>
+<em class=call> call all_vars_to_all_copies(ens_handle, label) </em>
+<pre>
+type(ensemble_type), intent(inout) :: <em class=code>ens_handle</em>
+character(len=*), intent(in), optional :: <em class=code>label</em>
+</pre>
+</div>
+
+<div class=indent1>
+<!-- Description -->
+
+<P>
+Transposes data from a representation in which each PE has a subset of
+copies of all variables to one in which each has all copies of a subset of
+variables. In the current implementation, storage is not released so both
+representations are always available. However, one representation may be
+current while the other is out of date.
+</P>
+<P>
+Different different numbers of copies, different lengths of the vectors,
+different numbers of PEs and different implementations of the MPI parallel
@@ Diff output truncated at 40000 characters. @@
More information about the Dart-dev
mailing list