[mpas-developers] Block Decomposition Revision

[Michael Duda]

Hi, Doug.

Thanks for sending along the updated document. I think we're headed in
the right direction, and I'm still digesting the proposed design. I did
have a couple of short questions, to begin with.

Is config_number_of_blocks the total number of blocks to be distributed
among all MPI tasks? If so, perhaps the default value could be some
special value, like 0, which would be interpreted as a request for one
block for each MPI task? This would make it easy to omit this option
from the namelist altogether except in cases where we would like to
assign multiple blocks to one or more MPI tasks.

In the proposed changes to mpas_block_decomp_cells_for_proc(...), I
think the existing argument partial_global_graph_info already supplies
what is essentially the cellsOnCell information, the idea (as you've
identified) being that we might decide to call some parallel online
graph partitioner that needs a description of the global graph
distributed among all of the calling processes. I think one argument for
passing a derived type, rather than just the cellsOnCell array, is that
we can extend this type to include, e.g., weights for each cell or edge.

Also, is the mpas_get_blocks_for_proc routine intended to be called by 
user code, or used internally by the block_decomp module (basically, is
it public or private)? If it's a public routine, I don't quite
understand why the blocks_per_proc argument is dimensioned by the number
of MPI tasks.

Perhaps as a general comment, it might be helpful to just describe the
public interfaces to the block_decomp module in terms of the meaning of
their input and output arguments, and leave the actual implementation
details out of the design section; this might help us in assessing
whether the interface meets the requirements without focusing too much
on the unimportant details of the implementation. Essentially, I think
more detail on the interfaces themselves would be helpful.

Cheers,
Michael


On Fri, Jan 27, 2012 at 10:55:55AM -0700, Doug Jacobsen wrote:
> Hi again all,
> 
> I have attached a newer updated version of the design doc, with several
> changes related to discussions I have had with various people. The changes
> should also address a lot of the concerns that were presented in previous
> emails about this topic.
> 
> I have also included some bigger picture information and general timeline
> information for the whole project ( all of tasks 1 through 5, not just this
> one).
> 
> Please let me know if you have any questions or concerns regarding this
> newly updated document.
> 
> Doug
> 
> On Thu, Jan 26, 2012 at 11:01 AM, Michael Duda <duda at ucar.edu> wrote:
> 
> > Hi, Doug.
> >
> > Thanks very much for the document. I'm still thinking through the design
> > section, but I have a couple of quick comments on the requirements.
> > First, I think we need to be careful about including implementation
> > details in the requirements themselves. For example, the requirement
> >
> >   "Domain decomposition information needs to be read from a file based on
> >    number of total blocks as opposed to number of MPI tasks."
> >
> > says quite a bit about how this should be implemented ("needs to be read
> > from a file"). Secondly, even though the initial design may take the
> > simplest approach to assigning multiple blocks to a task, I think it
> > will be good to ensure that the requirements describe our long-term
> > requirements for the module.
> >
> > Two of the requirements could be something along the lines of:
> >
> > 1) the user must be able to specify the number of blocks to be owned by
> >   each task; and
> >
> > 2) the block_decomp module must return information describing the cells
> >   contained in the specified number of blocks for the task.
> >
> > As for additional requirements, we could consider whether it is the case
> > that we ultimately want to be able to specify a different number of
> > blocks for each MPI task, and if we want to be able to place additional
> > constraints on which blocks are assigned to a task (much harder, I
> > think; e.g., preference should be given for contiguously located blocks
> > to maximize shared-memory copies, or for blocks that are spread as
> > evenly around the global domain as possible to help with load balancing).
> >
> > I apologize for taking so long to have a look through the document; I'll
> > send along any other comments that might come up as soon as I can.
> >
> > Regards,
> > Michael
> >
> >
> > On Tue, Jan 24, 2012 at 01:26:53PM -0700, Doug Jacobsen wrote:
> > > Hi All,
> > >
> > > Continuing with the work that Michale previously proposed before the
> > > derived data type revision, I am starting work on revisions to the block
> > > decomposition module to support multiple blocks per mpi process. As a
> > > refresher, this item was previously described as
> > >
> > > 4) Modify the block_decomp module to enable a task to get a list of
> > >     cells in more than one block that it is to be the owner of.
> > >     Implemented in the simplest way, there could simply be a namelist
> > >     option to specify how many blocks each task should own, and the
> > >     block_decomp module could look for a graph.info.part.n file, with
> > >     n=num_blocks_per_task*num_tasks, and assign blocks k, 2k, 3k, ...,
> > >     num_blocks_per_task*k to task k.
> > >
> > > I have attached a design document (pdf and latex) related to this task.
> > > Please let me know if you have any comments or suggestions. I would like
> > to
> > > have this implemented by the end of the week. Thanks for any feedback.
> > >
> > > Doug
> >
> >
> >
> > > _______________________________________________
> > > mpas-developers mailing list
> > > mpas-developers at mailman.ucar.edu
> > > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> >
> >



> _______________________________________________
> mpas-developers mailing list
> mpas-developers at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers