[mpas-developers] Block Decomposition Revision

Michael Duda duda at ucar.edu
Fri Jan 27 14:30:05 MST 2012


Hi, Doug.

Thanks for the quick response!

> I agree, I can remove the cellsOnCell from the input argument list. Should
> the extension of partial_global_graph_info be included in this design doc?
> Or should that be handled at a later time?

Since we don't currently have a way to define weights for cells or edges,
I'd not object to dealing with the extension of partial_global_graph_info
later, especially since it doesn't fundamentally affect the work of
assigning multiple blocks to an MPI task.

> The mpas_get_blocks_for_proc routine is going to be private, where there is
> a public vector called blocks_per_proc that contains the number of blocks
> that each processor has. That way if one processor wants to figure out how
> many blocks another has you can just look it up in that integer vector. The
> other method would be to input a processor number to this routine, and have
> the output be the number of blocks that processor has. Either way is fine
> with me, I just figured a lookup table would be more convenient for other
> parts of MPAS.

I'd not argue that it couldn't be useful for an MPI task to know how many
blocks another task owns, but I think we should consider whether the
mpas_get_blocks_per_proc() routine or the blocks_per_proc vector are
needed to meet any of the requirements. If not, do we want there to be a
requirement for such, and if so, why? Also, while recognizing that the
number of MPI tasks should be at most O(10^6) for the forseeable future, 
I think we should be wary of storing global information in memory.

> I only included the algorithms regarding the global_block_id mappings to
> local_block_id and owning_proc so that other people could examine them, and
> if there were any general issues with the mapping that I came up with. This
> was mainly because I figured everyone wasn't going to spend time reviewing
> the branch before it gets committed back to trunk, only you and I would
> really explore it much. But I can take some time and include more
> information on the interfaces themselves.

I think at some point in the past, the design document template may have
had a section named "Implementation"; so, maybe we could keep the
implementation details in this section, and focus on a detailed interface
specification in the Design section. Does that sound like a reasonable
option?

Regards,
Michael


On Fri, Jan 27, 2012 at 01:57:36PM -0700, Doug Jacobsen wrote:
> Hi Michael,
> 
> See Below.
> 
> On Fri, Jan 27, 2012 at 1:40 PM, Michael Duda <duda at ucar.edu> wrote:
> 
> > Hi, Doug.
> >
> > Thanks for sending along the updated document. I think we're headed in
> > the right direction, and I'm still digesting the proposed design. I did
> > have a couple of short questions, to begin with.
> >
> > Is config_number_of_blocks the total number of blocks to be distributed
> > among all MPI tasks? If so, perhaps the default value could be some
> > special value, like 0, which would be interpreted as a request for one
> > block for each MPI task? This would make it easy to omit this option
> > from the namelist altogether except in cases where we would like to
> > assign multiple blocks to one or more MPI tasks.
> >
> 
> That is correct. config_number_of_blocks should be the total number of
> blocks. I could easily create this option, which would simply default to
> the case where config_number_of_blocks = dminfo % nProcs. I did 1 just
> because that was the minimum number of blocks you would ever want to run
> with. But your idea seems better, and more inline with the current
> structure.
> 
> 
> > In the proposed changes to mpas_block_decomp_cells_for_proc(...), I
> > think the existing argument partial_global_graph_info already supplies
> > what is essentially the cellsOnCell information, the idea (as you've
> > identified) being that we might decide to call some parallel online
> > graph partitioner that needs a description of the global graph
> > distributed among all of the calling processes. I think one argument for
> > passing a derived type, rather than just the cellsOnCell array, is that
> > we can extend this type to include, e.g., weights for each cell or edge.
> >
> 
> I agree, I can remove the cellsOnCell from the input argument list. Should
> the extension of partial_global_graph_info be included in this design doc?
> Or should that be handled at a later time?
> 
> 
> > Also, is the mpas_get_blocks_for_proc routine intended to be called by
> > user code, or used internally by the block_decomp module (basically, is
> > it public or private)? If it's a public routine, I don't quite
> > understand why the blocks_per_proc argument is dimensioned by the number
> > of MPI tasks.
> >
> 
> The mpas_get_blocks_for_proc routine is going to be private, where there is
> a public vector called blocks_per_proc that contains the number of blocks
> that each processor has. That way if one processor wants to figure out how
> many blocks another has you can just look it up in that integer vector. The
> other method would be to input a processor number to this routine, and have
> the output be the number of blocks that processor has. Either way is fine
> with me, I just figured a lookup table would be more convenient for other
> parts of MPAS.
> 
> 
> > Perhaps as a general comment, it might be helpful to just describe the
> > public interfaces to the block_decomp module in terms of the meaning of
> > their input and output arguments, and leave the actual implementation
> > details out of the design section; this might help us in assessing
> > whether the interface meets the requirements without focusing too much
> > on the unimportant details of the implementation. Essentially, I think
> > more detail on the interfaces themselves would be helpful.
> >
> 
> I only included the algorithms regarding the global_block_id mappings to
> local_block_id and owning_proc so that other people could examine them, and
> if there were any general issues with the mapping that I came up with. This
> was mainly because I figured everyone wasn't going to spend time reviewing
> the branch before it gets committed back to trunk, only you and I would
> really explore it much. But I can take some time and include more
> information on the interfaces themselves.
> 
> Thanks for the comments,
> Doug
> 
> On Fri, Jan 27, 2012 at 10:55:55AM -0700, Doug Jacobsen wrote:
> > Hi again all,
> >
> > I have attached a newer updated version of the design doc, with several
> > changes related to discussions I have had with various people. The changes
> > should also address a lot of the concerns that were presented in previous
> > emails about this topic.
> >
> > I have also included some bigger picture information and general timeline
> > information for the whole project ( all of tasks 1 through 5, not just
> this
> > one).
> >
> > Please let me know if you have any questions or concerns regarding this
> > newly updated document.
> >
> > Doug
> >
> > On Thu, Jan 26, 2012 at 11:01 AM, Michael Duda <duda at ucar.edu> wrote:
> >
> > > Hi, Doug.
> > >
> > > Thanks very much for the document. I'm still thinking through the design
> > > section, but I have a couple of quick comments on the requirements.
> > > First, I think we need to be careful about including implementation
> > > details in the requirements themselves. For example, the requirement
> > >
> > >   "Domain decomposition information needs to be read from a file based
> on
> > >    number of total blocks as opposed to number of MPI tasks."
> > >
> > > says quite a bit about how this should be implemented ("needs to be read
> > > from a file"). Secondly, even though the initial design may take the
> > > simplest approach to assigning multiple blocks to a task, I think it
> > > will be good to ensure that the requirements describe our long-term
> > > requirements for the module.
> > >
> > > Two of the requirements could be something along the lines of:
> > >
> > > 1) the user must be able to specify the number of blocks to be owned by
> > >   each task; and
> > >
> > > 2) the block_decomp module must return information describing the cells
> > >   contained in the specified number of blocks for the task.
> > >
> > > As for additional requirements, we could consider whether it is the case
> > > that we ultimately want to be able to specify a different number of
> > > blocks for each MPI task, and if we want to be able to place additional
> > > constraints on which blocks are assigned to a task (much harder, I
> > > think; e.g., preference should be given for contiguously located blocks
> > > to maximize shared-memory copies, or for blocks that are spread as
> > > evenly around the global domain as possible to help with load
> balancing).
> > >
> > > I apologize for taking so long to have a look through the document; I'll
> > > send along any other comments that might come up as soon as I can.
> > >
> > > Regards,
> > > Michael
> > >
> > >
> > > On Tue, Jan 24, 2012 at 01:26:53PM -0700, Doug Jacobsen wrote:
> > > > Hi All,
> > > >
> > > > Continuing with the work that Michale previously proposed before the
> > > > derived data type revision, I am starting work on revisions to the
> block
> > > > decomposition module to support multiple blocks per mpi process. As a
> > > > refresher, this item was previously described as
> > > >
> > > > 4) Modify the block_decomp module to enable a task to get a list of
> > > >     cells in more than one block that it is to be the owner of.
> > > >     Implemented in the simplest way, there could simply be a namelist
> > > >     option to specify how many blocks each task should own, and the
> > > >     block_decomp module could look for a graph.info.part.n file, with
> > > >     n=num_blocks_per_task*num_tasks, and assign blocks k, 2k, 3k, ...,
> > > >     num_blocks_per_task*k to task k.
> > > >
> > > > I have attached a design document (pdf and latex) related to this
> task.
> > > > Please let me know if you have any comments or suggestions. I would
> like
> > > to
> > > > have this implemented by the end of the week. Thanks for any feedback.
> > > >
> > > > Doug
> > >
> > >
> > >
> > > > _______________________________________________
> > > > mpas-developers mailing list
> > > > mpas-developers at mailman.ucar.edu
> > > > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> > >
> > >
> 
> 
> 
> > _______________________________________________
> > mpas-developers mailing list
> > mpas-developers at mailman.ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/mpas-developers


More information about the mpas-developers mailing list