[Dart-dev] DART/branches Revision: 12425

[dart at ucar.edu]

nancy at ucar.edu
2018-03-06 10:53:49 -0700 (Tue, 06 Mar 2018)
487
this has notes and partially implmented code for putting
2 different additional distribution types into an ensemble type.
also an option on transpose_type 3 for replicating across
a subset of the total tasks.

this requires a group to be created in the mpi_utilities
and for the window manager to know which communicator to use.
can we make a group communicator that defaults to using all 
N tasks so the window manager can always just use the group
communicator for the ensemble mean?




Modified: DART/branches/rma_distrib/assimilation_code/modules/utilities/ensemble_manager_mod.f90
===================================================================
--- DART/branches/rma_distrib/assimilation_code/modules/utilities/ensemble_manager_mod.f90	2018-03-06 15:56:07 UTC (rev 12424)
+++ DART/branches/rma_distrib/assimilation_code/modules/utilities/ensemble_manager_mod.f90	2018-03-06 17:53:49 UTC (rev 12425)
@@ -4,6 +4,20 @@
 !
 ! $Id$
 
+!>@todo FIXME
+!> looking at design decisions to support distributed static data.
+!> also looking at how to support the quad filter layout.  it replicates
+!> each of the state vector items and then distributes pairs of items
+!> when going from var-complete to copy-complete.
+!>
+!> also looking at having multiple communication patterns; one the
+!> existing communicator which crosses all tasks and another which
+!> uses MPI groups to break up communication into something local to
+!> roughly node-sized (16-64 tasks/group).  this requires passing
+!> something down to the mpi calls (of which there are 7 used by this code)
+!> to indicate which communicator to use. (see the FIXME by the use mpi_utils
+!> line - there are 3 suggestions there for possible implementation paths.)
+
 module ensemble_manager_mod
 
 ! Manages a data structure that is composed of copies of a vector of variables.
@@ -19,13 +33,17 @@
                               error_handler, E_ERR, E_MSG, do_output, &
                               nmlfileunit, find_namelist_in_file,        &
                               check_namelist_read, timestamp, set_output
+use sort_mod,          only : index_sort
+use time_manager_mod,  only : time_type, set_time
 
-use time_manager_mod,  only : time_type, set_time
+!>@todo FIXME either: make group versions of each of these, overload them, or add an
+!> additional last argument saying this is a group call and not a world one
+
 use mpi_utilities_mod, only : task_count, my_task_id, send_to, receive_from, &
                               task_sync, broadcast_send, broadcast_recv
-use sort_mod,          only : index_sort
 
 
+
 implicit none
 private
 
@@ -56,69 +74,73 @@
 
 !>@todo update documentation with regard to 'single_restart_file_[in,out]'
 
-!>@todo FIXME the rule here should be that we only access %copies and %vars for efficiency
-!>but every other part of this structure should go through accessor routines.
+!>@todo FIXME the rule should be that we only access %copies and %vars for efficiency
+!> but every other part of this structure should go through accessor routines.
+!> we can leave only vars and copies public and set the rest of the entries private 
+!> by putting the 'private' attribute on all the other entries here.
 
-   !DIRECT ACCESS INTO STORAGE IS USED TO REDUCE COPYING: BE CAREFUL
-   !!!private
-   integer(i8)                  :: num_vars
+!> note that num_vars is an i8 but my_num_vars is not.  it's computationally more
+!> expensive to have long integers so we're trying to minimize where they happen.
+!> (we could make the option of redefining i8 to be i4 in types_mod.f90)
+!> we assume that no task will have the memory to allocate a full i8 buffer,
+!> so my_num_vars will remain expressible as an i4.
+
+>
+   integer(i8)                  :: num_vars 
    integer                      :: num_copies, my_num_copies, my_num_vars
    integer,        allocatable  :: my_copies(:)
    integer(i8),    allocatable  :: my_vars(:)
+   !DIRECT ACCESS INTO STORAGE IS USED TO REDUCE COPYING: BE CAREFUL
    ! Storage in next line is to be used when each pe has all copies of subset of vars
    real(r8),       allocatable  :: copies(:, :)         ! Dimensioned (num_copies, my_num_vars)
    ! Storage on next line is used when each pe has subset of copies of all vars
    real(r8),       allocatable  :: vars(:, :)           ! Dimensioned (num_vars, my_num_copies)
-   ! Time is only related to var complete
-   type(time_type), allocatable :: time(:)
+   type(time_type),         allocatable :: time(:)      ! Time is only related to var complete
+   type(time_type)              :: current_time         ! The current time, constant across the ensemble
    integer                      :: distribution_type
-   integer                      :: valid     ! copies modified last, vars modified last, both same
-   integer                      :: id_num
-   integer, allocatable         :: task_to_pe_list(:), pe_to_task_list(:) ! List of tasks
+   integer,private              :: valid     ! copies modified last, vars modified last, both same
+   integer,private              :: id_num
+   integer, allocatable,private :: task_to_pe_list(:), pe_to_task_list(:) ! List of tasks
    ! Flexible my_pe, layout_type which allows different task layouts for different ensemble handles
    integer                      :: my_pe
-   integer                      :: layout_type
-   integer                      :: transpose_type
-   integer                      :: num_extras
-   type(time_type)              :: current_time ! The current time, constant across the ensemble
+   integer,private              :: layout_type
+   integer,private              :: transpose_type
+   integer                      :: num_extras   ! FIXME: this needs to be more general.
 
 end type ensemble_type
 
+! FIXME: to separate filter use from plain storage issues,
+! if we can implement a class/flavor/category id, then we
+! can ask if an entry is of the given id, how many have that id, etc