Hi Michael,<br><br>Thanks for bringing this point up. It's a bit more subtle.<br><br>So, previously the get_owner_list routine has lived within the mpas_dmpar module. This was fine because the assumption was made that only one block per processor would be created, and the halo updates lived within mpas_dmpar. Also, get_owner_list required mpi_isend/mpi_irecv commands to determine who owns what cells, and how the cells should be exchanged.<br>
<br>The problem arises when you have to create exchange lists for linked lists of fields with multiple block information. As you can see from the interface I provided in the document there are two logical flags that tell if the owned/needed elements are decomposed or not. If they are not decomposed that means that each processor owns a chunk of them, which you could imagine would happen during the initial read in of fields prior to the first call of block_decomp where local_cell_list is created. If the elements are decomposed, that means that you now have a linked list of fields, where each element in the linked list represents the data associated with a block. In addition to this, that also means that this get_owner_list routine now has to call routines from the block_decomp module in order to determine who owns what block, and the local id of each block.<br>
<br>The circular dependency does not come about because of the mpas_block_creator module, it is actually one of the main reasons for the creation of the mpas_block_creator module. Currently there is a dependency within mpas_block_decomp that is uses mpas_dmpar. So the circular dependency is only present if we place the get_owner_list routine within either the mpas_block_decomp module or within the mpas_dmpar module. Since this routine requires mpas_block_decomp routines, it could be placed in that module, but it doesn't make sense with the other content of that module, and if we put it in mpas_dmpar then mpas_dmpar requires mpas_block_decomp, and mpas_block_decomp requires mpas_dmpar.<br>
<br>The only way I can see to remove MPI calls from an external module to mpas_dmpar is if we write out own intefaces for all of the mpi calls, like mpi_isend, mpi_irecv, and mpi_wait. But that seems like redundant work. However, if that's what we decide is the best course of action I'm not opposed to doing the work for that.<br>
<br>Thanks,<br>Doug<br><br><div class="gmail_quote">On Thu, May 3, 2012 at 6:32 PM, Michael Duda <span dir="ltr"><<a href="mailto:duda@ucar.edu" target="_blank">duda@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, Doug.<br>
<br>
I'm still pondering a few of the details in the new document, but one<br>
question that I still have concerns the last paragraph of the design<br>
chapter:<br>
<br>
One issue that comes up with the creation of this new module, is that<br>
MPI calls are now required within a module that’s external to mpas<br>
dmpar. Previously all MPI calls were isolated within the dmpar module,<br>
however because of some circular dependency issues that come up now this<br>
is no longer the case. Now, MPI calls can be restricted to the dmpar<br>
module and the block creation module.<br>
<br>
With the creation of the new module that you propose, mpas_block_creator,<br>
I'm not sure where the circular depencency comes in; could you say a bit<br>
more about this? A circular dependency would mean that mpas_dmpar is already<br>
using mpas_block_creator, hence module_block_creator cannot use mpas_dmpar<br>
and therefore needs to make its own direct MPI calls, right?<br>
<br>
Thanks,<br>
Michael<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Thu, May 03, 2012 at 11:45:16AM -0600, Doug Jacobsen wrote:<br>
> Hi All,<br>
><br>
> Ok here is my first try to make a diagram and include the interfaces<br>
> relevant within the framework.<br>
><br>
> Please let me know if this doesn't clear anything up. I'm not entirely sure<br>
> if this diagram is sufficient for the questions you asked, but if it's not<br>
> just let me know and I can work on it a bit more.<br>
><br>
> Thanks,<br>
> Doug<br>
<br>
<br>
<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> mpas-developers mailing list<br>
> <a href="mailto:mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br>
> <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers" target="_blank">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
<br>
</div></div></blockquote></div><br>