[mpas-developers] Block Decomposition Revision

[Doug Jacobsen]

Hi Michael,

See Below.

On Fri, Jan 27, 2012 at 1:40 PM, Michael Duda <duda at ucar.edu> wrote:

> Hi, Doug.
>
> Thanks for sending along the updated document. I think we're headed in
> the right direction, and I'm still digesting the proposed design. I did
> have a couple of short questions, to begin with.
>
> Is config_number_of_blocks the total number of blocks to be distributed
> among all MPI tasks? If so, perhaps the default value could be some
> special value, like 0, which would be interpreted as a request for one
> block for each MPI task? This would make it easy to omit this option
> from the namelist altogether except in cases where we would like to
> assign multiple blocks to one or more MPI tasks.
>

That is correct. config_number_of_blocks should be the total number of
blocks. I could easily create this option, which would simply default to
the case where config_number_of_blocks = dminfo % nProcs. I did 1 just
because that was the minimum number of blocks you would ever want to run
with. But your idea seems better, and more inline with the current
structure.


> In the proposed changes to mpas_block_decomp_cells_for_proc(...), I
> think the existing argument partial_global_graph_info already supplies
> what is essentially the cellsOnCell information, the idea (as you've
> identified) being that we might decide to call some parallel online
> graph partitioner that needs a description of the global graph
> distributed among all of the calling processes. I think one argument for
> passing a derived type, rather than just the cellsOnCell array, is that
> we can extend this type to include, e.g., weights for each cell or edge.
>

I agree, I can remove the cellsOnCell from the input argument list. Should
the extension of partial_global_graph_info be included in this design doc?
Or should that be handled at a later time?


> Also, is the mpas_get_blocks_for_proc routine intended to be called by
> user code, or used internally by the block_decomp module (basically, is
> it public or private)? If it's a public routine, I don't quite
> understand why the blocks_per_proc argument is dimensioned by the number
> of MPI tasks.
>

The mpas_get_blocks_for_proc routine is going to be private, where there is
a public vector called blocks_per_proc that contains the number of blocks
that each processor has. That way if one processor wants to figure out how
many blocks another has you can just look it up in that integer vector. The
other method would be to input a processor number to this routine, and have
the output be the number of blocks that processor has. Either way is fine
with me, I just figured a lookup table would be more convenient for other
parts of MPAS.


> Perhaps as a general comment, it might be helpful to just describe the
> public interfaces to the block_decomp module in terms of the meaning of
> their input and output arguments, and leave the actual implementation
> details out of the design section; this might help us in assessing
> whether the interface meets the requirements without focusing too much
> on the unimportant details of the implementation. Essentially, I think
> more detail on the interfaces themselves would be helpful.
>

I only included the algorithms regarding the global_block_id mappings to
local_block_id and owning_proc so that other people could examine them, and
if there were any general issues with the mapping that I came up with. This
was mainly because I figured everyone wasn't going to spend time reviewing
the branch before it gets committed back to trunk, only you and I would
really explore it much. But I can take some time and include more
information on the interfaces themselves.

Thanks for the comments,
Doug

On Fri, Jan 27, 2012 at 10:55:55AM -0700, Doug Jacobsen wrote:
> Hi again all,
>
> I have attached a newer updated version of the design doc, with several
> changes related to discussions I have had with various people. The changes
> should also address a lot of the concerns that were presented in previous
> emails about this topic.
>
> I have also included some bigger picture information and general timeline
> information for the whole project ( all of tasks 1 through 5, not just
this
> one).
>
> Please let me know if you have any questions or concerns regarding this
> newly updated document.
>
> Doug
>
> On Thu, Jan 26, 2012 at 11:01 AM, Michael Duda <duda at ucar.edu> wrote:
>
> > Hi, Doug.
> >
> > Thanks very much for the document. I'm still thinking through the design
> > section, but I have a couple of quick comments on the requirements.
> > First, I think we need to be careful about including implementation
> > details in the requirements themselves. For example, the requirement
> >
> >   "Domain decomposition information needs to be read from a file based
on
> >    number of total blocks as opposed to number of MPI tasks."
> >
> > says quite a bit about how this should be implemented ("needs to be read
> > from a file"). Secondly, even though the initial design may take the
> > simplest approach to assigning multiple blocks to a task, I think it
> > will be good to ensure that the requirements describe our long-term
> > requirements for the module.
> >
> > Two of the requirements could be something along the lines of:
> >
> > 1) the user must be able to specify the number of blocks to be owned by
> >   each task; and
> >
> > 2) the block_decomp module must return information describing the cells
> >   contained in the specified number of blocks for the task.
> >
> > As for additional requirements, we could consider whether it is the case
> > that we ultimately want to be able to specify a different number of
> > blocks for each MPI task, and if we want to be able to place additional
> > constraints on which blocks are assigned to a task (much harder, I
> > think; e.g., preference should be given for contiguously located blocks
> > to maximize shared-memory copies, or for blocks that are spread as
> > evenly around the global domain as possible to help with load
balancing).
> >
> > I apologize for taking so long to have a look through the document; I'll
> > send along any other comments that might come up as soon as I can.
> >
> > Regards,
> > Michael
> >
> >
> > On Tue, Jan 24, 2012 at 01:26:53PM -0700, Doug Jacobsen wrote:
> > > Hi All,
> > >
> > > Continuing with the work that Michale previously proposed before the
> > > derived data type revision, I am starting work on revisions to the
block
> > > decomposition module to support multiple blocks per mpi process. As a
> > > refresher, this item was previously described as
> > >
> > > 4) Modify the block_decomp module to enable a task to get a list of
> > >     cells in more than one block that it is to be the owner of.
> > >     Implemented in the simplest way, there could simply be a namelist
> > >     option to specify how many blocks each task should own, and the
> > >     block_decomp module could look for a graph.info.part.n file, with
> > >     n=num_blocks_per_task*num_tasks, and assign blocks k, 2k, 3k, ...,
> > >     num_blocks_per_task*k to task k.
> > >
> > > I have attached a design document (pdf and latex) related to this
task.
> > > Please let me know if you have any comments or suggestions. I would
like
> > to
> > > have this implemented by the end of the week. Thanks for any feedback.
> > >
> > > Doug
> >
> >
> >
> > > _______________________________________________
> > > mpas-developers mailing list
> > > mpas-developers at mailman.ucar.edu
> > > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> >
> >



> _______________________________________________
> mpas-developers mailing list
> mpas-developers at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120127/9779e64a/attachment.html