<p><b>dwj07@fsu.edu</b> 2012-05-02 09:33:23 -0600 (Wed, 02 May 2012)</p><p><br>
-- BRANCH COMMIT --<br>
<br>
Adding a design document for the implementation of multiple blocks.<br>
</p><hr noshade><pre><font color="gray">Added: branches/omp_blocks/docs/mpas_multiple_blocks.pdf
===================================================================
(Binary files differ)
Property changes on: branches/omp_blocks/docs/mpas_multiple_blocks.pdf
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: branches/omp_blocks/docs/mpas_multiple_blocks.tex
===================================================================
--- branches/omp_blocks/docs/mpas_multiple_blocks.tex (rev 0)
+++ branches/omp_blocks/docs/mpas_multiple_blocks.tex 2012-05-02 15:33:23 UTC (rev 1856)
@@ -0,0 +1,216 @@
+\documentclass[11pt]{report}
+
+\usepackage{graphicx}
+\usepackage{listings}
+\usepackage{color}
+
+\setlength{\topmargin}{0in}
+\setlength{\headheight}{0in}
+\setlength{\headsep}{0in}
+\setlength{\textheight}{9.0in}
+\setlength{\textwidth}{6.5in}
+\setlength{\evensidemargin}{0in}
+\setlength{\oddsidemargin}{0in}
+
+
+\begin{document}
+
+\title{Revisions to MPAS block decomposition routines}
+\author{}
+
+\maketitle
+\tableofcontents
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%
+% Introduction
+%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Introduction}
+
+Previously several projects have been worked on with the end goal of supporting
+multiple blocks. These tasks are described below.
+
+\begin{enumerate}
+
+\item Update/extend the fundamental derived types in mpas\_grid\_types.F.
+ In order for other parts of the infrastructure to handle multiple
+ blocks per task in a clean way, we'll need to be able to pass a head
+ pointer to a field into a routine, and have that routine loop through
+ all blocks for that field, with information about which cells/edges/vertices
+ in that field need to be communicated.
+
+\item Decide on a new MPAS I/O abstraction layer, which will provide a
+ high-level interface to the PIO layer for the rest of MPAS. This layer
+ should work with blocks of fields, and make it possible to define an
+ arbitrary set of I/O streams at run-time.
+
+\item Add a new module to parse a run-time I/O configuration file that
+ will describe which fields are read or written to each of the I/O
+ streams that a user requests via the file. This module will make calls
+ to the new MPAS I/O layer to register the requested fields for I/O in
+ the requested streams.
+
+\item Update the mpas\_dmpar module to support communication operations on
+ multiple blocks per task. This will likely involve revising the
+ internal data structures used to define communication of cells
+ between tasks, and also require revisions to the public interface
+ routines themselves.
+
+\item Modify the block\_decomp module to enable a task to get a list of
+ cells in more than one block that it is to be the owner of.
+ Implemented in the simplest way, there could simply be a namelist
+ option to specify how many blocks each task should own, and the
+ block\_decomp module could look for a graph.info.part.n file, with
+ n=num\_blocks\_per\_task*num\_tasks, and assign blocks k, 2k, 3k, ...,
+ num\_blocks\_per\_task*k to task k.
+
+\end{enumerate}
+
+This document related to tying all of these projects together, and allowing
+MPAS to run with multiple blocks per MPI process.
+
+As an example of what running with multiple blocks means, currently MPAS is
+allowed to run with 1 block per process. This means prior to running MPAS a
+decomposition of cells needs to be determined. This decomposition tells each
+MPI process which cells it owns. So, under the current framework the number of
+decompositions have to be equal to the number of MPI tasks used to run the job.
+
+After this project is completed, a user should be able to specify the use of a
+decomposition that does not have an equal number of blocks as the number of
+processors the job is run with. Typically this would be used to allow more than
+one block per processor, but could theoretically be used to run having some
+processors not have any blocks.
+
+The goal with this project is to allow exploration of OpenMP directives, and
+test performance using different block sizes. Allowing multiple blocks per
+processor could increase cache reuse and allow a performance increase.
+
+This work is currently being performed under
+branches/omp\_blocks/multiple\_blocks and all source code changes can be
+examined there.
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%
+% Requirements
+%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Requirements}
+
+There are significant changes to MPAS' framework that have to be made in order
+to support multiple blocks. A list of requirements that determine these changes
+are listed below.
+
+\begin{itemize}
+ \item Block creation must be robust, and handle an arbitrary number of
+ blocks per processor.
+
+ \item Blocks should be created using the derived data types created in an
+ earlier project, utilizing the field data types.
+
+ \item Block creation routines should be created with an arbitrary number of
+ halos assumed, although the default is currently two.
+
+ \item All block communication routines should be able to handle shared
+ memory copies.
+
+ \item Exchange list creation should be performed at the block/field level.
+
+ \item A new module should be setup to handle the management of blocks.
+\end{itemize}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%
+% Design
+%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Design}
+
+Only a small amount of design has been completed thus far. So, all information
+in this section should be regarded as a work in progress for now.
+
+The current prototyping efforts have determined the following routines which
+require changes to their infrastructure.
+
+\begin{lstlisting}[language=fortran,escapechar=@,frame=single]
+mpas_dmpar_alltoall_field
+mpas_dmpar_exch_halo_field
+mpas_dmpar_get_owner_list
+mpas_input_state_for_domain
+\end{lstlisting}
+
+Within the allToAll and exch\_halo routines, loops over multiple blocks need to
+be added where they are not currently in place. Also, shared memory copies
+within local blocks need to be added.
+
+The whole structure of mpas\_dmpar\_get\_owner\_list has to change in order to
+support multiple blocks. This routine currently builds the exchange lists for a
+single block and has global communications. In order to handle the creation of
+exchange lists with multiple blocks, this routine will be rewritten to use the
+field data types. This way each field will be a linked list, consisting of the
+fields from each block. The downside to this change, is that now the routine
+requires the setup of basic block types (really just the blockID number), and
+block fields (like indexToCellID) prior to calling this. However, this keeps
+the data types used in the creation of these routines in line with how the rest
+of MPAS deals with fields.
+
+mpas\_input\_state\_for\_domain also has to be modified in order to setup the
+fields and blocks as required for mpas\_dmpar\_get\_owner\_list, and to create
+multiple blocks. Currently it is writing under the assumption that only one
+block per process exists, and has that single block hard-coded as the only one
+that gets created.
+
+In addition to the currently in place changes that need to be made, several
+routines need to be added. Within the current prototyping work, a routine has
+been created to determine the all cell indices for a block, including all halo
+cells. It has been written under the assumption that there could be an
+arbitrary number of halos. This routine is called
+\begin{lstlisting}[language=fortran,escapechar=@,frame=single]
+mpas_get_halo_cells_and_exchange_lists(dminfo, nHalos,
+ indexToCellID_0Halo, nEdgesOnCell_0Halo,
+ cellsOnCell_0Halo, indexToCellID_nHalos,
+ nEdgesOnCell_nHalos, cellsOnCell_nHalos)
+\end{lstlisting}
+
+At the time of writing this document, this routine can be seen within the
+src/framework/mpas\_block\_decomp.F module, but this may change when a new
+module is created.
+
+One general change that has to be made in order to support these field data
+types being used in the input stage of MPAS is the addition of a deallocate
+field routine. This routine would be used to deallocate all fields within a
+field linked list. It is used when a field is created that's not a member of a
+block, so calling mpas\_deallocated\_block would not destroy all the memory
+associated with the field.
+
+In addition to the changes listed here, routines still need to be determined to
+create a list of vertices and edges for a block and all it's halos, as well as
+their respective exchange lists. After the list of cells, vertices, and edges
+are complete for a block the IO read fields can be called to setup the fields
+within each block. Finally, the global indices within a block need to be
+modified to be local indices.
+
+As mentioned in the requirements section, a large portion of these changes
+might be pushed into a new module. This new module would be written to handle
+the management of blocks. The proposed name would be mpas\_block\_manager.
+
+\chapter{Testing}
+The end goal from this project is to provide a framework that allows
+bit-for-bit reproduction of data using an arbitrary combination of blocks and
+processor numbers.
+
+Using this goal to define a testing strategy implies a good test would be
+exploring bit-for-bit reproduction of output data using the three following
+simulations:
+\begin{itemize}
+ \item Current trunk simulation run with 8 processors and 8 blocks
+ \item Finished branch simulation run with 8 processors and 8 blocks
+ \item Finished branch simulation run with 1 processor and 8 blocks
+ \item Finished branch simulation run with 2 processors and 8 blocks
+\end{itemize}
+
+If all of these simulations produce bit-for-bit output then the project would
+be deemed as completed.
+
+\end{document}
</font>
</pre>