Hi Conrad,<br><br>I don't know how against long verbose names everyone else is, but I would support something like haloLayersToExchange, or something more explicit like that. There are definitely some variable names that are of similar length, but I'm all for being as explicit as possible.<br>
<br>Doug<br><br><div class="gmail_quote">On Tue, Feb 21, 2012 at 8:53 PM, Ringler, Todd D <span dir="ltr"><<a href="mailto:ringler@lanl.gov">ringler@lanl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div><br>
</div>
Hi Conrad,
<div><br>
</div>
<div>Thanks for reply. It all looks great. haloIndices might cause confusion with others (like it did for me). haloLayers could certainly work. Regardless, it looks like an excellent bit of code.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Todd</div><div><div class="h5">
<div><br>
<div>
<div>On Feb 21, 2012, at 6:19 PM, Conrad Roesch wrote:</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">
<div>Hi Todd,</div>
<div><br>
</div>
<div>Thank you for your comments; I've included responses throughout the text below.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>conrad</div>
<div><br>
</div>
<div><br>
</div>
On Fri, Feb 17, 2012 at 08:28:33PM -0700, Todd Ringler wrote:<br>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">Hi Conrad,<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">This will be a great addition to the MPAS framework, so first off I<br>
</blockquote>
<blockquote type="cite">want to thank you all for pushing it along. Also, thanks for taking<br>
</blockquote>
<blockquote type="cite">the time to put this document together. Here are a couple of thoughts<br>
</blockquote>
<blockquote type="cite">.....<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">1. I might not be following it correctly, so feel free to point me in<br>
</blockquote>
<blockquote type="cite">the right direction here. It seems like the optional argument<br>
</blockquote>
<blockquote type="cite">(haloIndices) is a list of local {cell, vertex, edge} IDs that will<br>
</blockquote>
<blockquote type="cite">used to figure out what data to send/receive. This pushes a good bit<br>
</blockquote>
<blockquote type="cite">of the effort into the core, in the sense that the core has to<br>
</blockquote>
<blockquote type="cite">build/maintain/keep-track-off haloIndices of various sorts.<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">At init (discussed in #2 below) we might specify some integer<br>
</blockquote>
<blockquote type="cite">parameter like maxHaloWidth. This is currently two, but after this<br>
</blockquote>
<blockquote type="cite">design it would be set at runtime. Then at init the halo source code<br>
</blockquote>
<blockquote type="cite">will build (and store internally) haloIndices for {cells, edges,<br>
</blockquote>
<blockquote type="cite">vertices} for haloWidth from 1 to maxHaloWidth. Then the user calls<br>
</blockquote>
<blockquote type="cite">halo exchange with the optional argument is the requested width of the<br>
</blockquote>
<blockquote type="cite">halo update (with valid ranges from 1 to maxHaloWidth). This<br>
</blockquote>
<blockquote type="cite">essentially hides all of the mechanics inside halo exchange and gives<br>
</blockquote>
<blockquote type="cite">the developer an easy-to-understand optional argument.<br>
</blockquote>
<br>
Actually, the proposed design works in a similar way to what you've<br>
proposed in the second paragraph, though rather than the width of the<br>
halo to exchange, the haloIndices array specifies which layers will be<br>
exchanged; for example specifying haloIndices as (/1/) would exchange<br>
only the first layer of cells, while specifying (/1,2/) would exchange<br>
both the first and second layers; specifying (/2/) would exchange only<br>
the second layer. Perhaps the haloIndices argument would be better named<br>
haloLayers, or something similar, to indicate that it is the layers,<br>
rather than cell/edge/vertex indices, that are being specified?<br>
<br>
The initialization of the exchange lists for each of the halo layers is<br>
currently handled in the mpas_io_input module, and the plan is to<br>
re-implement this code in the same place, though in a more general way<br>
that would make it very simple to extend the number of halo layers that<br>
we can have for fields. However, I think that how we specify the number<br>
of halo layers (on a per-core basis, or a per-field basis?) might be a<br>
good subject for a separate design document; the intent of this<br>
proposal is to focus on the efficiency of halo exchanges in current<br>
cores by providing a mechanism for exchanging just the<br>
cells/edges/vertices that are needed by a particular loop.<br>
<br>
<blockquote type="cite">2. The design document discusses the actual call the halo update. How<br>
</blockquote>
<blockquote type="cite">will the process initialize? i.e. will there be an init interface?<br>
</blockquote>
<br>
As mentioned above, the plan is to replace the current halo<br>
initialization code in mpas_io_input with more modular and extensible<br>
code, where the number of halo layers is determined by a parameter in<br>
the code (right now, set to 2 for compatibility with current solver<br>
code). It would definitely make sense to subroutine-ize this new code to<br>
make it more modular, but a more flexible user interface would probably<br>
depend on how we decide to handle variable halo widths in future.<br>
<br>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">3. How will the design be tested? We have been considering the idea of<br>
</blockquote>
<blockquote type="cite">implementing a unit test for the MPI halo updates. As we move to tens<br>
</blockquote>
<blockquote type="cite">of thousands of processors, MPI-related stuff has the tendency to<br>
</blockquote>
<blockquote type="cite">break. Simply being able to easily test that the messages are<br>
</blockquote>
<blockquote type="cite">generated, sent and received correctly will likely prove to be a great<br>
</blockquote>
<blockquote type="cite">debugging tool as we move to bigger and bigger machines.<br>
</blockquote>
<br>
For initially testing correctness, it should suffice to create a dummy<br>
field in one of the MPAS cores, initialize all blocks of the field to<br>
some known value (say, 3.1416), and then place known garbage (say,<br>
-999.0) in all halo cells; then, we'll try exchanging different<br>
combinations of halo layers and confirming that exactly those layers are<br>
updated with correct (known) values. We can repeat this for dummy edge-<br>
and vertex-based fields as well.<br>
<br>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">4. I am guessing that the fieldType will contain meta data that will<br>
</blockquote>
<blockquote type="cite">allow haloExchange to figure out if it is updated cell, edge or vertex<br>
</blockquote>
<blockquote type="cite">data. Is this correct.<br>
</blockquote>
<br>
Correct. Given an array of halo layers to update, the MPAS field types<br>
have references to the correct exchange lists, which are correctly<br>
linked into the data type for cell-, edge-, and vertex-based fields.<br>
<br>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">Thanks again for sharing the document. I am excited to try out the<br>
</blockquote>
<blockquote type="cite">code!<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">Cheers, Todd<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">Begin forwarded message:<br>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">From: Conrad Roesch <<a href="mailto:croesch@ucar.edu" target="_blank">croesch@ucar.edu</a>> Subject: Halo Exchange Design<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">Document Date: February 17, 2012 11:15:34 AM MST To:<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><a href="mailto:mpas-developers@ucar.edu" target="_blank">mpas-developers@ucar.edu</a><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">Hello All,<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">I have drafted a design document (attached) that outlines some<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">proposed changes to the MPAS halo exchange routines. This work<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">would initially be done in a sub-branch of the omp_blocks branch of<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">the repository. Before implementing any changes, I wanted to give<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">everyone a chance to look this over, and so if anyone will require<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">modifications to this proposal, or have any suggestions or<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">questions, please let me know.<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">Cheers, Conrad<br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite"><br>
</blockquote>
</blockquote>
<blockquote type="cite"><br>
</blockquote>
<br>
<blockquote type="cite">_______________________________________________<br>
</blockquote>
<blockquote type="cite">mpas-developers mailing list<br>
</blockquote>
<blockquote type="cite"><a href="mailto:mpas-developers@mailman.ucar.edu" target="_blank">mpas-developers@mailman.ucar.edu</a><br>
</blockquote>
<blockquote type="cite"><a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers" target="_blank">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
</blockquote>
<div><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div></div></div>
<br>_______________________________________________<br>
mpas-developers mailing list<br>
<a href="mailto:mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br>
<a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers" target="_blank">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
<br></blockquote></div><br>