<p><b>dwj07@fsu.edu</b> 2012-01-24 13:29:11 -0700 (Tue, 24 Jan 2012)</p><p><br>
-- BRANCH COMMIT --<br>
<br>
Adding design document for redesign of mpas_block_decomp.<br>
</p><hr noshade><pre><font color="gray">Added: branches/omp_blocks/docs/mpas_block_decomp_redesign.pdf
===================================================================
(Binary files differ)
Property changes on: branches/omp_blocks/docs/mpas_block_decomp_redesign.pdf
___________________________________________________________________
Added: svn:mime-type
+ application/octet-stream
Added: branches/omp_blocks/docs/mpas_block_decomp_redesign.tex
===================================================================
--- branches/omp_blocks/docs/mpas_block_decomp_redesign.tex (rev 0)
+++ branches/omp_blocks/docs/mpas_block_decomp_redesign.tex 2012-01-24 20:29:11 UTC (rev 1416)
@@ -0,0 +1,179 @@
+\documentclass[11pt]{report}
+
+\usepackage{graphicx}
+\usepackage{listings}
+\usepackage{color}
+
+\setlength{\topmargin}{0in}
+\setlength{\headheight}{0in}
+\setlength{\headsep}{0in}
+\setlength{\textheight}{9.0in}
+\setlength{\textwidth}{6.5in}
+\setlength{\evensidemargin}{0in}
+\setlength{\oddsidemargin}{0in}
+
+
+\begin{document}
+
+\title{Revisions to MPAS block decomposition routines}
+\author{}
+
+\maketitle
+\tableofcontents
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%
+% Introduction
+%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Introduction}
+
+In order to support multiple blocks of cells per MPI task, there are a number of
+development issues that need to be addressed:
+
+\begin{enumerate}
+
+\item Update/extend the fundamental derived types in mpas\_grid\_types.F.
+ In order for other parts of the infrastructure to handle multiple
+ blocks per task in a clean way, we'll need to be able to pass a head
+ pointer to a field into a routine, and have that routine loop through
+ all blocks for that field, with information about which cells/edges/vertices
+ in that field need to be communicated.
+
+\item Decide on a new MPAS I/O abstraction layer, which will provide a
+ high-level interface to the PIO layer for the rest of MPAS. This layer
+ should work with blocks of fields, and make it possible to define an
+ arbitrary set of I/O streams at run-time.
+
+\item Add a new module to parse a run-time I/O configuration file that
+ will describe which fields are read or written to each of the I/O
+ streams that a user requests via the file. This module will make calls
+ to the new MPAS I/O layer to register the requested fields for I/O in
+ the requested streams.
+
+\item Update the mpas\_dmpar module to support communication operations on
+ multiple blocks per task. This will likely involve revising the
+ internal data structures used to define communication of cells
+ between tasks, and also require revisions to the public interface
+ routines themselves.
+
+\item Modify the block\_decomp module to enable a task to get a list of
+ cells in more than one block that it is to be the owner of.
+ Implemented in the simplest way, there could simply be a namelist
+ option to specify how many blocks each task should own, and the
+ block\_decomp module could look for a graph.info.part.n file, with
+ n=num\_blocks\_per\_task*num\_tasks, and assign blocks k, 2k, 3k, ...,
+ num\_blocks\_per\_task*k to task k.
+
+\end{enumerate}
+
+This document concerns the last item, namely, the extensions to the block decomposition
+module that will be necessary for supporting multiple blocks per task in other infrastructure
+modules.
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%
+% Requirements
+%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Requirements}
+
+The changes to the block decomposition module should enable an MPI task to get
+a list of it's owned cells, as well as the block number each of those cells
+lives on within it's task.
+
+\begin{itemize}
+
+ \item Domain decomposition information needs to be read from a file based
+ on number of total blocks as opposed to number of MPI tasks.
+
+ \item Block decomposition routines need to provide a list of cells in a on
+ a processor, as well as a list of blocks on a processor, and the cells
+ that make up each block.
+\end{itemize}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%
+% Design
+%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Design}
+
+We propose several changes to the block decomposition module in order to
+support multiple blocks per MPI task. Currently, in order to support the case
+where there are multiple blocks per MPI task a namelist parameter needs to be
+added that will allow these two values to differ. \\
+
+The namelist.input files will have the value config\_number\_of\_blocks added
+as an integer field to the io section. This option
+will have a default value of 1. \\
+
+Inside mpas\_block\_decomp.F, the mpas\_block\_decomp\_cells\_for\_proc needs
+to be changed. To not only read in all cells in all blocks, but also their
+block numbers. \\
+
+The meaning of the contents of graph.info.part.N needs to change from the
+processor ID that owns a cell, to the global block number for a cell. This
+means the file that is read in with have N = config\_number\_of\_blocks. \\
+
+Given a graph.info.part.N file, the global block number needs to be mapped into
+both an owning processor number, and a local block id. The local block id does
+not need to be computed within mpas\_block\_decomp\_cells\_for\_proc as long as
+the mapping is available or known. \\
+
+The api for mpas\_block\_decomp\_cells\_for\_proc will change from
+
+\begin{lstlisting}[language=fortran,escapechar=@,frame=single]
+@\small{subroutine mpas\_block\_decomp\_cells\_for\_proc(dminfo, partial\_global\_graph\_info, local\_cell\_list)}@
+\end{lstlisting}
+
+to
+
+\begin{lstlisting}[language=fortran,escapechar=@,frame=single]
+@\small{subroutine mpas\_block\_decomp\_cells\_for\_proc(dminfo, partial\_global\_graph\_info, local\_cell\_list, \colorbox{yellow}{local\_block\_list})}@
+\end{lstlisting}
+
+where local\_cell\_list is a list of cells owned by a processor, and
+local\_block\_list is a list of global block ids that describes which block
+owns a cell. \\
+
+mpas\_block\_decomp\_cells\_for\_proc will perform the same regardless of
+number of processors to enable the use of multiple blocks on a single
+processor. \\
+
+\pagebreak
+
+To begin, the mapping from global block id to owning processor number and local
+block number is as follows.
+
+\begin{lstlisting}[language=fortran,escapechar=@,frame=single]
+blocks_per_proc = config_number_of_blocks / dminfo % nprocs
+
+local_block_id = mod(global_block_id, blocks_per_proc)
+owning_proc = (global_block_id) / blocks_per_proc
+\end{lstlisting}
+
+An array called global\_block\_list needs to be added, which will be similar to
+global\_cell\_list except that instead of storing cell ids it will store global
+block ids that should be paired with the cell id. For example,
+global\_block\_list(10) gives the global block number that owns the cell from
+global\_cell\_list(10). \\
+
+An additional mpas\_dmpar\_scatter\_ints needs to be added to communicate the
+global\_block\_list into nprocs copies of local\_block\_list containing a list
+of the global block ids that a processor owns. \\
+
+%\begin{lstlisting}[language=fortran,escapechar=@,frame=single]
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%
+% Implementation
+%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\chapter{Implementation}
+
+Should we outline a plan for implementing these changes?
+
+\end{document}
</font>
</pre>