[mpas-developers] Halo Exchange Design Document

Tue Feb 21 18:19:20 MST 2012

Hi Todd,

Thank you for your comments; I've included responses throughout the text below.

Cheers,
conrad

On Fri, Feb 17, 2012 at 08:28:33PM -0700, Todd Ringler wrote:
> 
> Hi Conrad,
> 
> This will be a great addition to the MPAS framework, so first off I
> want to thank you all for pushing it along. Also, thanks for taking
> the time to put this document together. Here are a couple of thoughts
> .....
> 
> 1. I might not be following it correctly, so feel free to point me in
> the right direction here. It seems like the optional argument
> (haloIndices) is a list of local {cell, vertex, edge} IDs that will
> used to figure out what data to send/receive. This pushes a good bit
> of the effort into the core, in the sense that the core has to
> build/maintain/keep-track-off haloIndices of various sorts.
> 
> At init (discussed in #2 below) we might specify some integer
> parameter like maxHaloWidth. This is currently two, but after this
> design it would be set at runtime. Then at init the halo source code
> will build (and store internally) haloIndices for {cells, edges,
> vertices} for haloWidth from 1 to maxHaloWidth. Then the user calls
> halo exchange with the optional argument is the requested width of the
> halo update (with valid ranges from 1 to maxHaloWidth). This
> essentially hides all of the mechanics inside halo exchange and gives
> the developer an easy-to-understand optional argument.

Actually, the proposed design works in a similar way to what you've
proposed in the second paragraph, though rather than the width of the
halo to exchange, the haloIndices array specifies which layers will be
exchanged; for example specifying haloIndices as (/1/) would exchange
only the first layer of cells, while specifying (/1,2/) would exchange
both the first and second layers; specifying (/2/) would exchange only
the second layer. Perhaps the haloIndices argument would be better named
haloLayers, or something similar, to indicate that it is the layers,
rather than cell/edge/vertex indices, that are being specified?

The initialization of the exchange lists for each of the halo layers is
currently handled in the mpas_io_input module, and the plan is to
re-implement this code in the same place, though in a more general way
that would make it very simple to extend the number of halo layers that
we can have for fields. However, I think that how we specify the number
of halo layers (on a per-core basis, or a per-field basis?) might be a
good subject for a separate design document; the intent of this
proposal is to focus on the efficiency of halo exchanges in current
cores by providing a mechanism for exchanging just the
cells/edges/vertices that are needed by a particular loop.

> 2. The design document discusses the actual call the halo update. How
> will the process initialize? i.e. will there be an init interface?

As mentioned above, the plan is to replace the current halo
initialization code in mpas_io_input with more modular and extensible
code, where the number of halo layers is determined by a parameter in
the code (right now, set to 2 for compatibility with current solver
code). It would definitely make sense to subroutine-ize this new code to
make it more modular, but a more flexible user interface would probably
depend on how we decide to handle variable halo widths in future.

> 
> 3. How will the design be tested? We have been considering the idea of
> implementing a unit test for the MPI halo updates. As we move to tens
> of thousands of processors, MPI-related stuff has the tendency to
> break. Simply being able to easily test that the messages are
> generated, sent and received correctly will likely prove to be a great
> debugging tool as we move to bigger and bigger machines.

For initially testing correctness, it should suffice to create a dummy
field in one of the MPAS cores, initialize all blocks of the field to
some known value (say, 3.1416), and then place known garbage (say,
-999.0) in all halo cells; then, we'll try exchanging different
combinations of halo layers and confirming that exactly those layers are
updated with correct (known) values. We can repeat this for dummy edge-
and vertex-based fields as well.

> 
> 4. I am guessing that the fieldType will contain meta data that will
> allow haloExchange to figure out if it is updated cell, edge or vertex
> data. Is this correct.

Correct. Given an array of halo layers to update, the MPAS field types
have references to the correct exchange lists, which are correctly
linked into the data type for cell-, edge-, and vertex-based fields.

> 
> Thanks again for sharing the document. I am excited to try out the
> code!
> 
> Cheers, Todd
> 
> 
> Begin forwarded message:
> 
>> From: Conrad Roesch <croesch at ucar.edu> Subject: Halo Exchange Design
>> Document Date: February 17, 2012 11:15:34 AM MST To:
>> mpas-developers at ucar.edu
>> 
>> 
>> Hello All,
>> 
>> I have drafted a design document (attached) that outlines some
>> proposed changes to the MPAS halo exchange routines.  This work
>> would initially be done in a sub-branch of the omp_blocks branch of
>> the repository.  Before implementing any changes, I wanted to give
>> everyone a chance to look this over, and so if anyone will require
>> modifications to this proposal, or have any suggestions or
>> questions, please let me know.
>> 
>> Cheers, Conrad
>> 
>> 
>> 
> 

> _______________________________________________
> mpas-developers mailing list
> mpas-developers at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120221/898b2523/attachment.html 