[mpas-developers] Multiple Blocks Design Document

[Jones, Philip W]

All,

I have to admit that the current doc is still kinda confusing to me right now.   Some of this is because decisions touch a lot of different aspects of the infrastructure, but some is just due to the infrastructure itself becoming a bit jumbled.  And some is because the doc is currently focused on specific tasks rather than overall design.

So...I think a couple of things would help.

First, do we have a layout of the architecture (esp. At these lower levels)?  That might help to see whether we need to think about reorganizing some stuff in the framework.  And help to see the dependencies and how we might limit the dependencies and hide more of the detail.  A nice figure showing  the modules, the dependencies, etc. would be great (doesn’t have to be UML – just a quick sketch).

Second, it might help to see how the main data structures are going to change by showing, say, the field and block structures.

Third, some pseudo code showing where the block loops will sit and how the info is passed might help to see whether support for arrays can still happen.

Finally, some interfaces for functions like halo updates and/or reductions would be useful to see how we can minimize what’s exposed at higher levels.

Right now, I guess I still have a muddied picture on how this looks in MPAS.

Thanks,

Phil


On 5/3/12 9:38 AM, "Doug Jacobsen" <jacobsen.douglas at gmail.com> wrote:

Hi Michael,

Here is a new version of the design document where I tried to incorporate your comments, and my responses.

One comment that you made that I'm not sure how to address is the following:

"Maybe we could add something about having user-level interfaces that minimize the amount of redundant information that must be passed to them, effectively ruling out designs that use simple arrays and exchange lists in user-level interfaces in favor of interfaces that work with derived types."

Maybe if you could clarify what you mean I could extend the requirement to incorporate this comment as well. Please let me know if there are any other issues.

Thanks,
Doug

On Wed, May 2, 2012 at 6:46 PM, Doug Jacobsen <jacobsen.douglas at gmail.com> wrote:
Hi Michael,

Here are some responses to your comments.

MGD: What does it mean for block creation to be ’robust’ ? Will we support
zero blocks per processor?

In this case, I think we should support zero blocks per processor. All of the cores should easily be able to support this is we just do something like nullify(domain % blocklist). This way, we could potentially provide a block decomposition where some processors are used for other tasks. Robust in this case means to me that any number of blocks could be used for any number of processors. Even if numBlocks < numProcs.

MGD: Does this imply that simple arrays will not be used in
any of the block decomp routines? If simple arrays will be used in some cases,
what will distinguish those cases from those that use field types?

>From the work that I have been doing so far, it seems infeasible to support simple arrays for most of the cases we are interested in. Granted, I might be missing something, but it seems like the only real way we can handle the required things is to use the field derived data types, and create "ad-hoc" blocks, with the required information. So, personally I would lean towards using the derived data types where at all possible, rather than relying on simple arrays in most cases.

MGD: Will we support different numbers of halo
layers on a per-field basis, or will we require that all fields and blocks have the
same number of halo layers?

We have had some discussions about this within the ocean group. The best case would be if we could eventually support different halo layers per field. However, regardless of how many fields have a given number of halos, the grid information is required to contain the maximum number of halos for any given field in state. So, we could potentially add a variable within the field data type called "maxHaloLayer" or something like that, and ensure that data is never accessed in a halo layer outside the maxHaloLayer for a given field. Though this is theoretically possible, I think it makes more sense as it's own project to determine how best to approach this issue. So I guess this requirement is mostly stated to allow the possibility of different fields having different halos, but more simply to allow all fields to have an undetermined number of halos.

MGD: Is this really a requirement?

Todd actually had this same point, and suggested I change the requirement to something like "All code related to the management of blocks should be isolated from the remainder of MPAS".

MGD: Generally, it seems to me like a couple of the above are really design issues ...

I agree with the comment regarding trying to phase out simple arrays in favor of derived data types.

MGD: See notes below — I think a lot of the work currently in mpas input state for domain should be migrated elsewhere.

I agree again, I'm currently trying to move a lot of this work into the other module, as discussed earlier. Currently I still have it in place, but it was mostly just for the prototyping phase to make sure I understood how everything was working.

MGD: It’s not clear to me from the argument list how exchange-list information is returned; ...

This is a more subtle issue that took me a while to decide on. This is also the issue that lead me to work with derived data types rather than simple arrays. Each block requires it's own set of exchange lists (send, recv, copy). Since the routines that create the exchange lists require information about which cells are owned by which processors, and which blocks those cells are in it is no longer possible to send in a pointer to a send/recv/copy list and have it output in a sensible fashion. We would not longer be getting exchange lists for the block owned by a processor, it would be a whole list of exchange lists for all blocks the processor owns.

So, to actually describe how the routine works. Instead of outputting the exchange lists, after each exchange list is created it is stored in field_ptr % block % cellsToSend(iHalo). If anyone has a better idea of how to do this, I'm open to ideas. However, this was the only way I could think of to actually implement it.

MGD: I think this work might be a good opportunity to better organize the parts ...

I agree again. This would be a good opportunity to encapsulate the routines better and try to make it easier to understand what is happening within the IO routines.

The only issue I have noticed with the new IO layer is that it handles simple arrays rather than field data types, as I mentioned to you earlier.

Also, I understand the preference to have all MPI calls limited to within the dmpar module. However, it seems like some code *can't* be put in that routine. For example, the routine which creates exchange lists. In the case of multiple blocks, this routine now needs to have access to the mpas_block_decomp module. However, the mpas_block_decomp module needs to have access to the mpas_dmpar module. This routine also requires MPI_Isend and MPI_Irecv to determine which processor owns which elements for the exchange list. Placing this routine within mpas_dmpar creates a circular dependency between the two modules.

It might be better to extend the MPI restriction to a few modules, like mpas_dmpar and mpas_block_manager (or whatever name we decide on). However, the most lax I want to make this requirement is all modules in framework.

Thanks for your suggestions and comments. Hopefully we can decide on a good infrastructure to support multiple blocks that will be easily extensible in the future.

Doug










On Wed, May 2, 2012 at 3:57 PM, Michael Duda <duda at ucar.edu> wrote:
Hi, Doug.

Thanks very much for starting the discussion. I've attached a version
of the document with a few comments and questions.

Michael


On Wed, May 02, 2012 at 09:32:53AM -0600, Doug Jacobsen wrote:
> Hi All,
>
> I've been working on implementing the necessary changes within MPAS'
> framework to allow the use of multiple blocks. These changes are rather far
> reaching, and require a lot of design choices. I have implemented working
> prototypes of several aspects of this project so far, but have not created
> a working model.
>
> At this point in the implementation process, I think I have a good idea
> what all has to change in order to support multiple blocks. In order to
> make sure that my design ideas are in line with everyone else's assumptions
> about how blocks should be created I put together a quick design document.
> I have attached both the tex source, and the pdf of this design document.
> Please keep in mind while reading this that is it a rather rough design
> document so far, and I created it mostly to get the conversation going to
> ensure everyone is happy with the end product.
>
> I left out the actual source changes that I have made so far for two
> reasons. First they are rather large (in line number), and they touch
> several files. If anyone is interested in looking at the actual code for
> specific routines please don't hesitate to ask, and I can reference you to
> the file with the changes in it. All of the work is being performed under
> branches/omp_blocks/mulitple_blocks (as is referenced in the design
> document). So, feel free to have a look at the code. Please note however,
> that it is a work in progress and by no means complete. But it can give you
> an idea of how I was planning everything to work.
>
> I am also going to commit this design document to branches/omp_blocks/docs
> so, if you want to read it from there feel free.
>
> Please feel free to provide comments and questions in any way you would
> like.
>
> Thanks,
> Doug



> _______________________________________________
> mpas-developers mailing list
> mpas-developers at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers




________________________________
_______________________________________________
mpas-developers mailing list
mpas-developers at mailman.ucar.edu
http://mailman.ucar.edu/mailman/listinfo/mpas-developers


---
Correspondence/TSPA/DUSA EARTH
------------------------------------------------------------
Philip Jones                                pwjones at lanl.gov
Climate, Ocean and Sea Ice Modeling
Los Alamos National Laboratory
T-3 MS B216                                 Ph: 505-500-2699
PO Box 1663                                Fax: 505-665-5926
Los Alamos, NM 87545-1663



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120503/eda69724/attachment.html