[mpas-developers] Halo Exchange Design Document

[Ringler, Todd D]

Hi Conrad,

Thanks for reply. It all looks great. haloIndices might cause confusion with others (like it did for me). haloLayers could certainly work. Regardless, it looks like an excellent bit of code.

Cheers,
Todd

On Feb 21, 2012, at 6:19 PM, Conrad Roesch wrote:

Hi Todd,

Thank you for your comments; I've included responses throughout the text below.

Cheers,
conrad


On Fri, Feb 17, 2012 at 08:28:33PM -0700, Todd Ringler wrote:

Hi Conrad,

This will be a great addition to the MPAS framework, so first off I
want to thank you all for pushing it along. Also, thanks for taking
the time to put this document together. Here are a couple of thoughts
.....

1. I might not be following it correctly, so feel free to point me in
the right direction here. It seems like the optional argument
(haloIndices) is a list of local {cell, vertex, edge} IDs that will
used to figure out what data to send/receive. This pushes a good bit
of the effort into the core, in the sense that the core has to
build/maintain/keep-track-off haloIndices of various sorts.

At init (discussed in #2 below) we might specify some integer
parameter like maxHaloWidth. This is currently two, but after this
design it would be set at runtime. Then at init the halo source code
will build (and store internally) haloIndices for {cells, edges,
vertices} for haloWidth from 1 to maxHaloWidth. Then the user calls
halo exchange with the optional argument is the requested width of the
halo update (with valid ranges from 1 to maxHaloWidth). This
essentially hides all of the mechanics inside halo exchange and gives
the developer an easy-to-understand optional argument.

Actually, the proposed design works in a similar way to what you've
proposed in the second paragraph, though rather than the width of the
halo to exchange, the haloIndices array specifies which layers will be
exchanged; for example specifying haloIndices as (/1/) would exchange
only the first layer of cells, while specifying (/1,2/) would exchange
both the first and second layers; specifying (/2/) would exchange only
the second layer. Perhaps the haloIndices argument would be better named
haloLayers, or something similar, to indicate that it is the layers,
rather than cell/edge/vertex indices, that are being specified?

The initialization of the exchange lists for each of the halo layers is
currently handled in the mpas_io_input module, and the plan is to
re-implement this code in the same place, though in a more general way
that would make it very simple to extend the number of halo layers that
we can have for fields. However, I think that how we specify the number
of halo layers (on a per-core basis, or a per-field basis?) might be a
good subject for a separate design document; the intent of this
proposal is to focus on the efficiency of halo exchanges in current
cores by providing a mechanism for exchanging just the
cells/edges/vertices that are needed by a particular loop.

2. The design document discusses the actual call the halo update. How
will the process initialize? i.e. will there be an init interface?

As mentioned above, the plan is to replace the current halo
initialization code in mpas_io_input with more modular and extensible
code, where the number of halo layers is determined by a parameter in
the code (right now, set to 2 for compatibility with current solver
code). It would definitely make sense to subroutine-ize this new code to
make it more modular, but a more flexible user interface would probably
depend on how we decide to handle variable halo widths in future.


3. How will the design be tested? We have been considering the idea of
implementing a unit test for the MPI halo updates. As we move to tens
of thousands of processors, MPI-related stuff has the tendency to
break. Simply being able to easily test that the messages are
generated, sent and received correctly will likely prove to be a great
debugging tool as we move to bigger and bigger machines.

For initially testing correctness, it should suffice to create a dummy
field in one of the MPAS cores, initialize all blocks of the field to
some known value (say, 3.1416), and then place known garbage (say,
-999.0) in all halo cells; then, we'll try exchanging different
combinations of halo layers and confirming that exactly those layers are
updated with correct (known) values. We can repeat this for dummy edge-
and vertex-based fields as well.


4. I am guessing that the fieldType will contain meta data that will
allow haloExchange to figure out if it is updated cell, edge or vertex
data. Is this correct.

Correct. Given an array of halo layers to update, the MPAS field types
have references to the correct exchange lists, which are correctly
linked into the data type for cell-, edge-, and vertex-based fields.


Thanks again for sharing the document. I am excited to try out the
code!

Cheers, Todd


Begin forwarded message:

From: Conrad Roesch <croesch at ucar.edu<mailto:croesch at ucar.edu>> Subject: Halo Exchange Design
Document Date: February 17, 2012 11:15:34 AM MST To:
mpas-developers at ucar.edu<mailto:mpas-developers at ucar.edu>


Hello All,

I have drafted a design document (attached) that outlines some
proposed changes to the MPAS halo exchange routines.  This work
would initially be done in a sub-branch of the omp_blocks branch of
the repository.  Before implementing any changes, I wanted to give
everyone a chance to look this over, and so if anyone will require
modifications to this proposal, or have any suggestions or
questions, please let me know.

Cheers, Conrad





_______________________________________________
mpas-developers mailing list
mpas-developers at mailman.ucar.edu<mailto:mpas-developers at mailman.ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/mpas-developers


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120222/2117a2f3/attachment-0001.html