[mpas-developers] Halo Exchange Design Document

[Doug Jacobsen]

Hi Conrad,

I don't know how against long verbose names everyone else is, but I would
support something like haloLayersToExchange, or something more explicit
like that. There are definitely some variable names that are of similar
length, but I'm all for being as explicit as possible.

Doug

On Tue, Feb 21, 2012 at 8:53 PM, Ringler, Todd D <ringler at lanl.gov> wrote:

>
>  Hi Conrad,
>
>  Thanks for reply. It all looks great. haloIndices might cause confusion
> with others (like it did for me). haloLayers could certainly work.
> Regardless, it looks like an excellent bit of code.
>
>  Cheers,
> Todd
>
>  On Feb 21, 2012, at 6:19 PM, Conrad Roesch wrote:
>
>  Hi Todd,
>
>  Thank you for your comments; I've included responses throughout the text
> below.
>
>  Cheers,
> conrad
>
>
>  On Fri, Feb 17, 2012 at 08:28:33PM -0700, Todd Ringler wrote:
>
>
>  Hi Conrad,
>
>
>  This will be a great addition to the MPAS framework, so first off I
>
> want to thank you all for pushing it along. Also, thanks for taking
>
> the time to put this document together. Here are a couple of thoughts
>
> .....
>
>
>  1. I might not be following it correctly, so feel free to point me in
>
> the right direction here. It seems like the optional argument
>
> (haloIndices) is a list of local {cell, vertex, edge} IDs that will
>
> used to figure out what data to send/receive. This pushes a good bit
>
> of the effort into the core, in the sense that the core has to
>
> build/maintain/keep-track-off haloIndices of various sorts.
>
>
>  At init (discussed in #2 below) we might specify some integer
>
> parameter like maxHaloWidth. This is currently two, but after this
>
> design it would be set at runtime. Then at init the halo source code
>
> will build (and store internally) haloIndices for {cells, edges,
>
> vertices} for haloWidth from 1 to maxHaloWidth. Then the user calls
>
> halo exchange with the optional argument is the requested width of the
>
> halo update (with valid ranges from 1 to maxHaloWidth). This
>
> essentially hides all of the mechanics inside halo exchange and gives
>
> the developer an easy-to-understand optional argument.
>
>
> Actually, the proposed design works in a similar way to what you've
> proposed in the second paragraph, though rather than the width of the
> halo to exchange, the haloIndices array specifies which layers will be
> exchanged; for example specifying haloIndices as (/1/) would exchange
> only the first layer of cells, while specifying (/1,2/) would exchange
> both the first and second layers; specifying (/2/) would exchange only
> the second layer. Perhaps the haloIndices argument would be better named
> haloLayers, or something similar, to indicate that it is the layers,
> rather than cell/edge/vertex indices, that are being specified?
>
> The initialization of the exchange lists for each of the halo layers is
> currently handled in the mpas_io_input module, and the plan is to
> re-implement this code in the same place, though in a more general way
> that would make it very simple to extend the number of halo layers that
> we can have for fields. However, I think that how we specify the number
> of halo layers (on a per-core basis, or a per-field basis?) might be a
> good subject for a separate design document; the intent of this
> proposal is to focus on the efficiency of halo exchanges in current
> cores by providing a mechanism for exchanging just the
> cells/edges/vertices that are needed by a particular loop.
>
> 2. The design document discusses the actual call the halo update. How
>
> will the process initialize? i.e. will there be an init interface?
>
>
> As mentioned above, the plan is to replace the current halo
> initialization code in mpas_io_input with more modular and extensible
> code, where the number of halo layers is determined by a parameter in
> the code (right now, set to 2 for compatibility with current solver
> code). It would definitely make sense to subroutine-ize this new code to
> make it more modular, but a more flexible user interface would probably
> depend on how we decide to handle variable halo widths in future.
>
>
>  3. How will the design be tested? We have been considering the idea of
>
> implementing a unit test for the MPI halo updates. As we move to tens
>
> of thousands of processors, MPI-related stuff has the tendency to
>
> break. Simply being able to easily test that the messages are
>
> generated, sent and received correctly will likely prove to be a great
>
> debugging tool as we move to bigger and bigger machines.
>
>
> For initially testing correctness, it should suffice to create a dummy
> field in one of the MPAS cores, initialize all blocks of the field to
> some known value (say, 3.1416), and then place known garbage (say,
> -999.0) in all halo cells; then, we'll try exchanging different
> combinations of halo layers and confirming that exactly those layers are
> updated with correct (known) values. We can repeat this for dummy edge-
> and vertex-based fields as well.
>
>
>  4. I am guessing that the fieldType will contain meta data that will
>
> allow haloExchange to figure out if it is updated cell, edge or vertex
>
> data. Is this correct.
>
>
> Correct. Given an array of halo layers to update, the MPAS field types
> have references to the correct exchange lists, which are correctly
> linked into the data type for cell-, edge-, and vertex-based fields.
>
>
>  Thanks again for sharing the document. I am excited to try out the
>
> code!
>
>
>  Cheers, Todd
>
>
>
>  Begin forwarded message:
>
>
>  From: Conrad Roesch <croesch at ucar.edu> Subject: Halo Exchange Design
>
>  Document Date: February 17, 2012 11:15:34 AM MST To:
>
>  mpas-developers at ucar.edu
>
>
>
>   Hello All,
>
>
>   I have drafted a design document (attached) that outlines some
>
>  proposed changes to the MPAS halo exchange routines.  This work
>
>  would initially be done in a sub-branch of the omp_blocks branch of
>
>  the repository.  Before implementing any changes, I wanted to give
>
>  everyone a chance to look this over, and so if anyone will require
>
>  modifications to this proposal, or have any suggestions or
>
>  questions, please let me know.
>
>
>   Cheers, Conrad
>
>
>
>
>
>
> _______________________________________________
>
> mpas-developers mailing list
>
> mpas-developers at mailman.ucar.edu
>
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers
>
>
>
>
> _______________________________________________
> mpas-developers mailing list
> mpas-developers at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120221/070677b4/attachment.html