<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Hi Todd,</div><div><br></div><div>Thank you for your comments; I've included responses throughout the text below.</div><div><br></div><div>Cheers,</div><div>conrad</div><div><br></div><div><br></div>On Fri, Feb 17, 2012 at 08:28:33PM -0700, Todd Ringler wrote:<br><blockquote type="cite"><br></blockquote><blockquote type="cite">Hi Conrad,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">This will be a great addition to the MPAS framework, so first off I<br></blockquote><blockquote type="cite">want to thank you all for pushing it along. Also, thanks for taking<br></blockquote><blockquote type="cite">the time to put this document together. Here are a couple of thoughts<br></blockquote><blockquote type="cite">.....<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">1. I might not be following it correctly, so feel free to point me in<br></blockquote><blockquote type="cite">the right direction here. It seems like the optional argument<br></blockquote><blockquote type="cite">(haloIndices) is a list of local {cell, vertex, edge} IDs that will<br></blockquote><blockquote type="cite">used to figure out what data to send/receive. This pushes a good bit<br></blockquote><blockquote type="cite">of the effort into the core, in the sense that the core has to<br></blockquote><blockquote type="cite">build/maintain/keep-track-off haloIndices of various sorts.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">At init (discussed in #2 below) we might specify some integer<br></blockquote><blockquote type="cite">parameter like maxHaloWidth. This is currently two, but after this<br></blockquote><blockquote type="cite">design it would be set at runtime. Then at init the halo source code<br></blockquote><blockquote type="cite">will build (and store internally) haloIndices for {cells, edges,<br></blockquote><blockquote type="cite">vertices} for haloWidth from 1 to maxHaloWidth. Then the user calls<br></blockquote><blockquote type="cite">halo exchange with the optional argument is the requested width of the<br></blockquote><blockquote type="cite">halo update (with valid ranges from 1 to maxHaloWidth). This<br></blockquote><blockquote type="cite">essentially hides all of the mechanics inside halo exchange and gives<br></blockquote><blockquote type="cite">the developer an easy-to-understand optional argument.<br></blockquote><br>Actually, the proposed design works in a similar way to what you've<br>proposed in the second paragraph, though rather than the width of the<br>halo to exchange, the haloIndices array specifies which layers will be<br>exchanged; for example specifying haloIndices as (/1/) would exchange<br>only the first layer of cells, while specifying (/1,2/) would exchange<br>both the first and second layers; specifying (/2/) would exchange only<br>the second layer. Perhaps the haloIndices argument would be better named<br>haloLayers, or something similar, to indicate that it is the layers,<br>rather than cell/edge/vertex indices, that are being specified?<br><br>The initialization of the exchange lists for each of the halo layers is<br>currently handled in the mpas_io_input module, and the plan is to<br>re-implement this code in the same place, though in a more general way<br>that would make it very simple to extend the number of halo layers that<br>we can have for fields. However, I think that how we specify the number<br>of halo layers (on a per-core basis, or a per-field basis?) might be a<br>good subject for a separate design document; the intent of this<br>proposal is to focus on the efficiency of halo exchanges in current<br>cores by providing a mechanism for exchanging just the<br>cells/edges/vertices that are needed by a particular loop.<br><br><blockquote type="cite">2. The design document discusses the actual call the halo update. How<br></blockquote><blockquote type="cite">will the process initialize? i.e. will there be an init interface?<br></blockquote><br>As mentioned above, the plan is to replace the current halo<br>initialization code in mpas_io_input with more modular and extensible<br>code, where the number of halo layers is determined by a parameter in<br>the code (right now, set to 2 for compatibility with current solver<br>code). It would definitely make sense to subroutine-ize this new code to<br>make it more modular, but a more flexible user interface would probably<br>depend on how we decide to handle variable halo widths in future.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">3. How will the design be tested? We have been considering the idea of<br></blockquote><blockquote type="cite">implementing a unit test for the MPI halo updates. As we move to tens<br></blockquote><blockquote type="cite">of thousands of processors, MPI-related stuff has the tendency to<br></blockquote><blockquote type="cite">break. Simply being able to easily test that the messages are<br></blockquote><blockquote type="cite">generated, sent and received correctly will likely prove to be a great<br></blockquote><blockquote type="cite">debugging tool as we move to bigger and bigger machines.<br></blockquote><br>For initially testing correctness, it should suffice to create a dummy<br>field in one of the MPAS cores, initialize all blocks of the field to<br>some known value (say, 3.1416), and then place known garbage (say,<br>-999.0) in all halo cells; then, we'll try exchanging different<br>combinations of halo layers and confirming that exactly those layers are<br>updated with correct (known) values. We can repeat this for dummy edge-<br>and vertex-based fields as well.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">4. I am guessing that the fieldType will contain meta data that will<br></blockquote><blockquote type="cite">allow haloExchange to figure out if it is updated cell, edge or vertex<br></blockquote><blockquote type="cite">data. Is this correct.<br></blockquote><br>Correct. Given an array of halo layers to update, the MPAS field types<br>have references to the correct exchange lists, which are correctly<br>linked into the data type for cell-, edge-, and vertex-based fields.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">Thanks again for sharing the document. I am excited to try out the<br></blockquote><blockquote type="cite">code!<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Cheers, Todd<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Begin forwarded message:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">From: Conrad Roesch <<a href="mailto:croesch@ucar.edu">croesch@ucar.edu</a>> Subject: Halo Exchange Design<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Document Date: February 17, 2012 11:15:34 AM MST To:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:mpas-developers@ucar.edu">mpas-developers@ucar.edu</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Hello All,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I have drafted a design document (attached) that outlines some<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">proposed changes to the MPAS halo exchange routines. This work<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">would initially be done in a sub-branch of the omp_blocks branch of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the repository. Before implementing any changes, I wanted to give<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">everyone a chance to look this over, and so if anyone will require<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">modifications to this proposal, or have any suggestions or<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">questions, please let me know.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Cheers, Conrad<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><br><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">mpas-developers mailing list<br></blockquote><blockquote type="cite"><a href="mailto:mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br></blockquote><blockquote type="cite"><a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br></blockquote><div><br></div></body></html>