[mpas-developers] Multiple Blocks Design Document

Wed May 2 18:46:41 MDT 2012

Hi Michael,

Here are some responses to your comments.

MGD: What does it mean for block creation to be ’robust’ ? Will we support
zero blocks per processor?

In this case, I think we should support zero blocks per processor. All of
the cores should easily be able to support this is we just do something
like nullify(domain % blocklist). This way, we could potentially provide a
block decomposition where some processors are used for other tasks. Robust
in this case means to me that any number of blocks could be used for any
number of processors. Even if numBlocks < numProcs.

MGD: Does this imply that simple arrays will not be used in
any of the block decomp routines? If simple arrays will be used in some
cases,
what will distinguish those cases from those that use field types?

>From the work that I have been doing so far, it seems infeasible to support
simple arrays for most of the cases we are interested in. Granted, I might
be missing something, but it seems like the only real way we can handle the
required things is to use the field derived data types, and create "ad-hoc"
blocks, with the required information. So, personally I would lean towards
using the derived data types where at all possible, rather than relying on
simple arrays in most cases.

MGD: Will we support different numbers of halo
layers on a per-field basis, or will we require that all fields and blocks
have the
same number of halo layers?

We have had some discussions about this within the ocean group. The best
case would be if we could eventually support different halo layers per
field. However, regardless of how many fields have a given number of halos,
the grid information is required to contain the maximum number of halos for
any given field in state. So, we could potentially add a variable within
the field data type called "maxHaloLayer" or something like that, and
ensure that data is never accessed in a halo layer outside the maxHaloLayer
for a given field. Though this is theoretically possible, I think it makes
more sense as it's own project to determine how best to approach this
issue. So I guess this requirement is mostly stated to allow the
possibility of different fields having different halos, but more simply to
allow all fields to have an undetermined number of halos.

MGD: Is this really a requirement?

Todd actually had this same point, and suggested I change the requirement
to something like "All code related to the management of blocks should be
isolated from the remainder of MPAS".

MGD: Generally, it seems to me like a couple of the above are really design
issues ...

I agree with the comment regarding trying to phase out simple arrays in
favor of derived data types.

MGD: See notes below — I think a lot of the work currently in mpas input
state for domain should be migrated elsewhere.

I agree again, I'm currently trying to move a lot of this work into the
other module, as discussed earlier. Currently I still have it in place, but
it was mostly just for the prototyping phase to make sure I understood how
everything was working.

MGD: It’s not clear to me from the argument list how exchange-list
information is returned; ...

This is a more subtle issue that took me a while to decide on. This is also
the issue that lead me to work with derived data types rather than simple
arrays. Each block requires it's own set of exchange lists (send, recv,
copy). Since the routines that create the exchange lists require
information about which cells are owned by which processors, and which
blocks those cells are in it is no longer possible to send in a pointer to
a send/recv/copy list and have it output in a sensible fashion. We would
not longer be getting exchange lists for the block owned by a processor, it
would be a whole list of exchange lists for all blocks the processor owns.

So, to actually describe how the routine works. Instead of outputting the
exchange lists, after each exchange list is created it is stored in
field_ptr % block % cellsToSend(iHalo). If anyone has a better idea of how
to do this, I'm open to ideas. However, this was the only way I could think
of to actually implement it.

MGD: I think this work might be a good opportunity to better organize the
parts ...

I agree again. This would be a good opportunity to encapsulate the routines
better and try to make it easier to understand what is happening within the
IO routines.

The only issue I have noticed with the new IO layer is that it handles
simple arrays rather than field data types, as I mentioned to you earlier.

Also, I understand the preference to have all MPI calls limited to within
the dmpar module. However, it seems like some code *can't* be put in that
routine. For example, the routine which creates exchange lists. In the case
of multiple blocks, this routine now needs to have access to the
mpas_block_decomp module. However, the mpas_block_decomp module needs to
have access to the mpas_dmpar module. This routine also requires MPI_Isend
and MPI_Irecv to determine which processor owns which elements for the
exchange list. Placing this routine within mpas_dmpar creates a circular
dependency between the two modules.

It might be better to extend the MPI restriction to a few modules, like
mpas_dmpar and mpas_block_manager (or whatever name we decide on). However,
the most lax I want to make this requirement is all modules in framework.

Thanks for your suggestions and comments. Hopefully we can decide on a good
infrastructure to support multiple blocks that will be easily extensible in
the future.

Doug

On Wed, May 2, 2012 at 3:57 PM, Michael Duda <duda at ucar.edu> wrote:

> Hi, Doug.
>
> Thanks very much for starting the discussion. I've attached a version
> of the document with a few comments and questions.
>
> Michael
>
>
> On Wed, May 02, 2012 at 09:32:53AM -0600, Doug Jacobsen wrote:
> > Hi All,
> >
> > I've been working on implementing the necessary changes within MPAS'
> > framework to allow the use of multiple blocks. These changes are rather
> far
> > reaching, and require a lot of design choices. I have implemented working
> > prototypes of several aspects of this project so far, but have not
> created
> > a working model.
> >
> > At this point in the implementation process, I think I have a good idea
> > what all has to change in order to support multiple blocks. In order to
> > make sure that my design ideas are in line with everyone else's
> assumptions
> > about how blocks should be created I put together a quick design
> document.
> > I have attached both the tex source, and the pdf of this design document.
> > Please keep in mind while reading this that is it a rather rough design
> > document so far, and I created it mostly to get the conversation going to
> > ensure everyone is happy with the end product.
> >
> > I left out the actual source changes that I have made so far for two
> > reasons. First they are rather large (in line number), and they touch
> > several files. If anyone is interested in looking at the actual code for
> > specific routines please don't hesitate to ask, and I can reference you
> to
> > the file with the changes in it. All of the work is being performed under
> > branches/omp_blocks/mulitple_blocks (as is referenced in the design
> > document). So, feel free to have a look at the code. Please note however,
> > that it is a work in progress and by no means complete. But it can give
> you
> > an idea of how I was planning everything to work.
> >
> > I am also going to commit this design document to
> branches/omp_blocks/docs
> > so, if you want to read it from there feel free.
> >
> > Please feel free to provide comments and questions in any way you would
> > like.
> >
> > Thanks,
> > Doug
>
>
>
> > _______________________________________________
> > mpas-developers mailing list
> > mpas-developers at mailman.ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120502/bfc52cc4/attachment.html 