[mpas-developers] Multiple Blocks Design Document

Thu May 3 09:38:09 MDT 2012

Hi Michael,

Here is a new version of the design document where I tried to incorporate
your comments, and my responses.

One comment that you made that I'm not sure how to address is the following:

"Maybe we could add something about having user-level interfaces that
minimize the amount of redundant information that must be passed to them,
effectively ruling out designs that use simple arrays and exchange lists in
user-level interfaces in favor of interfaces that work with derived types."

Maybe if you could clarify what you mean I could extend the requirement to
incorporate this comment as well. Please let me know if there are any other
issues.

Thanks,
Doug

On Wed, May 2, 2012 at 6:46 PM, Doug Jacobsen <jacobsen.douglas at gmail.com>wrote:

> Hi Michael,
>
> Here are some responses to your comments.
>
> MGD: What does it mean for block creation to be ’robust’ ? Will we support
> zero blocks per processor?
>
> In this case, I think we should support zero blocks per processor. All of
> the cores should easily be able to support this is we just do something
> like nullify(domain % blocklist). This way, we could potentially provide a
> block decomposition where some processors are used for other tasks. Robust
> in this case means to me that any number of blocks could be used for any
> number of processors. Even if numBlocks < numProcs.
>
> MGD: Does this imply that simple arrays will not be used in
> any of the block decomp routines? If simple arrays will be used in some
> cases,
> what will distinguish those cases from those that use field types?
>
> From the work that I have been doing so far, it seems infeasible to
> support simple arrays for most of the cases we are interested in. Granted,
> I might be missing something, but it seems like the only real way we can
> handle the required things is to use the field derived data types, and
> create "ad-hoc" blocks, with the required information. So, personally I
> would lean towards using the derived data types where at all possible,
> rather than relying on simple arrays in most cases.
>
> MGD: Will we support different numbers of halo
> layers on a per-field basis, or will we require that all fields and blocks
> have the
> same number of halo layers?
>
> We have had some discussions about this within the ocean group. The best
> case would be if we could eventually support different halo layers per
> field. However, regardless of how many fields have a given number of halos,
> the grid information is required to contain the maximum number of halos for
> any given field in state. So, we could potentially add a variable within
> the field data type called "maxHaloLayer" or something like that, and
> ensure that data is never accessed in a halo layer outside the maxHaloLayer
> for a given field. Though this is theoretically possible, I think it makes
> more sense as it's own project to determine how best to approach this
> issue. So I guess this requirement is mostly stated to allow the
> possibility of different fields having different halos, but more simply to
> allow all fields to have an undetermined number of halos.
>
> MGD: Is this really a requirement?
>
> Todd actually had this same point, and suggested I change the requirement
> to something like "All code related to the management of blocks should be
> isolated from the remainder of MPAS".
>
> MGD: Generally, it seems to me like a couple of the above are really
> design issues ...
>
> I agree with the comment regarding trying to phase out simple arrays in
> favor of derived data types.
>
> MGD: See notes below — I think a lot of the work currently in mpas input
> state for domain should be migrated elsewhere.
>
> I agree again, I'm currently trying to move a lot of this work into the
> other module, as discussed earlier. Currently I still have it in place, but
> it was mostly just for the prototyping phase to make sure I understood how
> everything was working.
>
> MGD: It’s not clear to me from the argument list how exchange-list
> information is returned; ...
>
> This is a more subtle issue that took me a while to decide on. This is
> also the issue that lead me to work with derived data types rather than
> simple arrays. Each block requires it's own set of exchange lists (send,
> recv, copy). Since the routines that create the exchange lists require
> information about which cells are owned by which processors, and which
> blocks those cells are in it is no longer possible to send in a pointer to
> a send/recv/copy list and have it output in a sensible fashion. We would
> not longer be getting exchange lists for the block owned by a processor, it
> would be a whole list of exchange lists for all blocks the processor owns.
>
> So, to actually describe how the routine works. Instead of outputting the
> exchange lists, after each exchange list is created it is stored in
> field_ptr % block % cellsToSend(iHalo). If anyone has a better idea of how
> to do this, I'm open to ideas. However, this was the only way I could think
> of to actually implement it.
>
> MGD: I think this work might be a good opportunity to better organize the
> parts ...
>
> I agree again. This would be a good opportunity to encapsulate the
> routines better and try to make it easier to understand what is happening
> within the IO routines.
>
> The only issue I have noticed with the new IO layer is that it handles
> simple arrays rather than field data types, as I mentioned to you earlier.
>
> Also, I understand the preference to have all MPI calls limited to within
> the dmpar module. However, it seems like some code *can't* be put in that
> routine. For example, the routine which creates exchange lists. In the case
> of multiple blocks, this routine now needs to have access to the
> mpas_block_decomp module. However, the mpas_block_decomp module needs to
> have access to the mpas_dmpar module. This routine also requires MPI_Isend
> and MPI_Irecv to determine which processor owns which elements for the
> exchange list. Placing this routine within mpas_dmpar creates a circular
> dependency between the two modules.
>
> It might be better to extend the MPI restriction to a few modules, like
> mpas_dmpar and mpas_block_manager (or whatever name we decide on). However,
> the most lax I want to make this requirement is all modules in framework.
>
> Thanks for your suggestions and comments. Hopefully we can decide on a
> good infrastructure to support multiple blocks that will be easily
> extensible in the future.
>
> Doug
>
>
>
>
>
>
>
>
>
>
> On Wed, May 2, 2012 at 3:57 PM, Michael Duda <duda at ucar.edu> wrote:
>
>> Hi, Doug.
>>
>> Thanks very much for starting the discussion. I've attached a version
>> of the document with a few comments and questions.
>>
>> Michael
>>
>>
>> On Wed, May 02, 2012 at 09:32:53AM -0600, Doug Jacobsen wrote:
>> > Hi All,
>> >
>> > I've been working on implementing the necessary changes within MPAS'
>> > framework to allow the use of multiple blocks. These changes are rather
>> far
>> > reaching, and require a lot of design choices. I have implemented
>> working
>> > prototypes of several aspects of this project so far, but have not
>> created
>> > a working model.
>> >
>> > At this point in the implementation process, I think I have a good idea
>> > what all has to change in order to support multiple blocks. In order to
>> > make sure that my design ideas are in line with everyone else's
>> assumptions
>> > about how blocks should be created I put together a quick design
>> document.
>> > I have attached both the tex source, and the pdf of this design
>> document.
>> > Please keep in mind while reading this that is it a rather rough design
>> > document so far, and I created it mostly to get the conversation going
>> to
>> > ensure everyone is happy with the end product.
>> >
>> > I left out the actual source changes that I have made so far for two
>> > reasons. First they are rather large (in line number), and they touch
>> > several files. If anyone is interested in looking at the actual code for
>> > specific routines please don't hesitate to ask, and I can reference you
>> to
>> > the file with the changes in it. All of the work is being performed
>> under
>> > branches/omp_blocks/mulitple_blocks (as is referenced in the design
>> > document). So, feel free to have a look at the code. Please note
>> however,
>> > that it is a work in progress and by no means complete. But it can give
>> you
>> > an idea of how I was planning everything to work.
>> >
>> > I am also going to commit this design document to
>> branches/omp_blocks/docs
>> > so, if you want to read it from there feel free.
>> >
>> > Please feel free to provide comments and questions in any way you would
>> > like.
>> >
>> > Thanks,
>> > Doug
>>
>>
>>
>> > _______________________________________________
>> > mpas-developers mailing list
>> > mpas-developers at mailman.ucar.edu
>> > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120503/99e08cf8/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpas_multiple_blocks.tex
Type: application/x-tex
Size: 14072 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120503/99e08cf8/attachment-0001.tex 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpas_multiple_blocks.pdf
Type: application/pdf
Size: 53924 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120503/99e08cf8/attachment-0001.pdf 