[mpas-developers] Multiple Blocks Design Document

[Doug Jacobsen]

Hi Michael,

Thanks for bringing this point up. It's a bit more subtle.

So, previously the get_owner_list routine has lived within the mpas_dmpar
module. This was fine because the assumption was made that only one block
per processor would be created, and the halo updates lived within
mpas_dmpar. Also, get_owner_list required mpi_isend/mpi_irecv commands to
determine who owns what cells, and how the cells should be exchanged.

The problem arises when you have to create exchange lists for linked lists
of fields with multiple block information. As you can see from the
interface I provided in the document there are two logical flags that tell
if the owned/needed elements are decomposed or not. If they are not
decomposed that means that each processor owns a chunk of them, which you
could imagine would happen during the initial read in of fields prior to
the first call of block_decomp where local_cell_list is created. If the
elements are decomposed, that means that you now have a linked list of
fields, where each element in the linked list represents the data
associated with a block. In addition to this, that also means that this
get_owner_list routine now has to call routines from the block_decomp
module in order to determine who owns what block, and the local id of each
block.

The circular dependency does not come about because of the
mpas_block_creator module, it is actually one of the main reasons for the
creation of the mpas_block_creator module. Currently there is a dependency
within mpas_block_decomp that is uses mpas_dmpar. So the circular
dependency is only present if we place the get_owner_list routine within
either the mpas_block_decomp module or within the mpas_dmpar module. Since
this routine requires mpas_block_decomp routines, it could be placed in
that module, but it doesn't make sense with the other content of that
module, and if we put it in mpas_dmpar then mpas_dmpar requires
mpas_block_decomp, and mpas_block_decomp requires mpas_dmpar.

The only way I can see to remove MPI calls from an external module to
mpas_dmpar is if we write out own intefaces for all of the mpi calls, like
mpi_isend, mpi_irecv, and mpi_wait. But that seems like redundant work.
However, if that's what we decide is the best course of action I'm not
opposed to doing the work for that.

Thanks,
Doug

On Thu, May 3, 2012 at 6:32 PM, Michael Duda <duda at ucar.edu> wrote:

> Hi, Doug.
>
> I'm still pondering a few of the details in the new document, but one
> question that I still have concerns the last paragraph of the design
> chapter:
>
>   One issue that comes up with the creation of this new module, is that
>   MPI calls are now required within a module that’s external to mpas
>   dmpar. Previously all MPI calls were isolated within the dmpar module,
>   however because of some circular dependency issues that come up now this
>   is no longer the case. Now, MPI calls can be restricted to the dmpar
>   module and the block creation module.
>
> With the creation of the new module that you propose, mpas_block_creator,
> I'm not sure where the circular depencency comes in; could you say a bit
> more about this? A circular dependency would mean that mpas_dmpar is
> already
> using mpas_block_creator, hence module_block_creator cannot use mpas_dmpar
> and therefore needs to make its own direct MPI calls, right?
>
> Thanks,
> Michael
>
>
> On Thu, May 03, 2012 at 11:45:16AM -0600, Doug Jacobsen wrote:
> > Hi All,
> >
> > Ok here is my first try to make a diagram and include the interfaces
> > relevant within the framework.
> >
> > Please let me know if this doesn't clear anything up. I'm not entirely
> sure
> > if this diagram is sufficient for the questions you asked, but if it's
> not
> > just let me know and I can work on it a bit more.
> >
> > Thanks,
> > Doug
>
>
>
>
> > _______________________________________________
> > mpas-developers mailing list
> > mpas-developers at mailman.ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120503/6f59a805/attachment.html