<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>Re: [mpas-developers] Block Decomposition Revision</title>
</head>
<body>
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt"><br>
Guys,<br>
<br>
I guess the local block ID was useful to us in previous implementations both for packing/unpacking messages and for indexing into arrays since we had an array block index rather than a linked list. But you can certainly do this with only a global id, the local
index just saves a search step. In the end, I think this is a relatively simple change that we can either add/remove later.<br>
<br>
Phil<br>
<br>
<br>
On 1/30/12 11:23 AM, "Doug Jacobsen" <<a href="jacobsen.douglas@gmail.com">jacobsen.douglas@gmail.com</a>> wrote:<br>
<br>
</span></font>
<blockquote><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt">Hi Michael,<br>
<br>
</span></font>
<blockquote><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt">I was thinking a bit more about the design document this weekend, and, with<br>
the blocks stored in a linked-list, I'm wondering whether we really need<br>
local block indices. The main use of local indices that comes to my mind<br>
would be in indexing arrays of blocks, though I'm not sure whether we'll<br>
need to do that; perhaps we will optimize the shared-memory copies using<br>
arrays of block pointers? Adding a bit of extra code to deal with local<br>
indices seems straightforward enough and might be useful in future, but the<br>
overall design might be simpler if blocks were just identified by a unique,<br>
global ID.<br>
</span></font></blockquote>
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt"><br>
I'm not sure how useful the local block id inside of the block_type would be, I just figured it might be helpful to have. Especially if we had to swap local blocks with id's 1 between two processors later. It might be easier to determine which blocks we are
talking about, especially when their global block id's wouldn't directly map to those local block id's anymore. Of course, for the short term, and the goals of this project we would not be tackling that problem and it wouldn't really be of any use yet. It
was most of a long term goal that I was trying to prepare for, with Phil's mention of dynamic load balancing.
<br>
<br>
</span></font>
<blockquote><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt">Also, would it improve usability of the mpas_block_decomp_cells_for_proc()<br>
interface if we returned the block ownership information (i.e., local_block_list)<br>
in a different form? For example, it's not specified which order, if any,<br>
the cells in local_cell_list will be returned (alternatively, there's no<br>
guarantee that local_block_list will be in sorted order); so, perhaps we<br>
can always return cells that are in the same block contiguously in<br>
local_cell_list, and then provide block information with arrays<br>
<br>
block_id(:) - global IDs of the blocks<br>
NB: size(block_id) = number of blocks owned by the task<br>
<br>
block_start(:) - offset in local_cell_list of the first cell in the blocks<br>
<br>
block_count(:) - number of cells in the blocks<br>
<br>
The above is obviously just one alternative to local_block_list, but it<br>
might make it easier for the caller to quickly determine how many blocks it<br>
is being given, the sizes of those blocks, and the global cell IDs in each<br>
of those blocks.<br>
</span></font></blockquote>
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt"><br>
So, in this case local_cell_list would remain the same, but it would be sorted by local_block_id right? I could see this method being easier for the IO module to handle, since it can easily build one block at a time rather than having to loop over both local_cell_list
and local_block_list to determine which cells are in a specific block. It's really just a matter of if we want to sort the structures inside of the mpas_block_decomp_cells_for_proc call, or after the data is returned. Either one is fine with me, so since you
are tackling the IO rewrite if you would rather push this into the block_decomp modules I can take care of that.<br>
<br>
Thanks for the comments,<br>
Doug<br>
<br>
<br>
On Fri, Jan 27, 2012 at 06:49:59PM -0700, Doug Jacobsen wrote:<br>
> Thanks for the comments Phil,<br>
><br>
> I fixed the typo you mentioned, and I have already changed the design<br>
> document to only provide a public interface to the<br>
> mpas_get_blocks_per_proc, rather than the lookup table.<br>
><br>
> Considering dynamic load balancing, would it be useful to store the local<br>
> block id, and owning processor number within the block derived data type? I<br>
> could see at least the local block id being useful, to help you if you need<br>
> to rearrange the block indices, or insert a new block into the correct<br>
> position, but I am unsure about the owning proc information. I guess that<br>
> would just be dminfo % my_proc_id, since you would only have blocks that<br>
> you would own.<br>
><br>
> If anyone has any input on these ideas please let me know, but for now I'll<br>
> add the local block index to the block data type (in the design doc), and<br>
> leave out the owning proc id.<br>
><br>
> Thanks again,<br>
> Doug<br>
><br>
> On Fri, Jan 27, 2012 at 3:52 PM, Jones, Philip W <<a href="pwjones@lanl.gov">pwjones@lanl.gov</a>> wrote:<br>
><br>
> ><br>
> > Doug,<br>
> ><br>
</span></font>
<blockquote><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt">> > Generally, I think it?s ok. Couple of minor things below. Michael is<br>
> > correct in saying we want to be careful about holding onto arrays that are<br>
> > proportional to either number of tasks/blocks, even if they appear small<br>
> > now. Memory per node is not growing and on a per-core basis will continue<br>
> > to shrink. I suspect we will only need these functions on<br>
> > start-up/initialization and may be able to deallocate after this phase is<br>
> > complete (and then the functions would return an error). Once we?ve set up<br>
> > communicators/patterns for halos, reductions and I/O, I don?t believe we?ll<br>
> > have a need for these sorts of global maps. Even if we did dynamic load<br>
> > balancing, we?d typically be passing off a block or two from one task to<br>
> > another and can update locally without these global queries. Maybe we can<br>
> > keep these around for now and worry about this later.<br>
> ><br>
> > For minor issues:<br>
> ><br>
> > Do we want this in the io namelist block? I think there should be a<br>
> > general decomposition group rather than putting this in i/o. (I have other<br>
> > naming preferences too, but that?ll open another can o? worms).<br>
> ><br>
> > The last para on page 6 of the pdf is missing something ?wher is the<br>
> > number of proc?. It appears in the tex input too, so it?s not a tex issue.<br>
> ><br>
> > Phil<br>
> ><br>
> ><br>
> ><br>
> > On 1/27/12 2:53 PM, "Doug Jacobsen" <<a href="jacobsen.douglas@gmail.com">jacobsen.douglas@gmail.com</a>> wrote:<br>
> ><br>
> > Hi again all,<br>
> ><br>
> > I have now attached the newest version of this design doc.<br>
> ><br>
> > Please let me know of any other comments/questions.<br>
> ><br>
> > Doug<br>
> ><br>
> > ------------------------------<br>
> > _______________________________________________<br>
> > mpas-developers mailing list<br>
> > <a href="mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br>
> > <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
> ><br>
> ><br>
> ><br>
> > ---<br>
> > Correspondence/TSPA/DUSA EARTH<br>
> > ------------------------------------------------------------<br>
> > Philip Jones <a href="pwjones@lanl.gov">pwjones@lanl.gov</a><br>
> > Climate, Ocean and Sea Ice Modeling<br>
> > Los Alamos National Laboratory<br>
> > T-3 MS B216 Ph: 505-500-2699 <tel:505-500-2699>
<br>
> > PO Box 1663 Fax: 505-665-5926 <tel:505-665-5926>
<br>
> > Los Alamos, NM 87545-1663<br>
> ><br>
> ><br>
> ><br>
> ><br>
<br>
> _______________________________________________<br>
> mpas-developers mailing list<br>
> <a href="mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br>
> <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
<br>
</span></font></blockquote>
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt"><br>
<br>
<br>
<hr align="CENTER" size="3" width="95%">
</span></font><span style="font-size:12pt"><font face="Courier, Courier New">_______________________________________________<br>
mpas-developers mailing list<br>
<a href="mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br>
<a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
</font></span></blockquote>
<span style="font-size:12pt"><font face="Courier, Courier New"><br>
<br>
---<br>
Correspondence/TSPA/DUSA EARTH<br>
------------------------------------------------------------<br>
Philip Jones <a href="pwjones@lanl.gov">pwjones@lanl.gov</a><br>
Climate, Ocean and Sea Ice Modeling<br>
Los Alamos National Laboratory<br>
T-3 MS B216 Ph: 505-500-2699<br>
PO Box 1663 Fax: 505-665-5926<br>
Los Alamos, NM 87545-1663<br>
</font></span><font size="1"><font face="Helvetica, Verdana, Arial"><span style="font-size:9pt"><br>
<br>
</span></font></font><font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:12pt"><br>
</span></font>
</body>
</html>