Hi Michael and Phil,<br><br>Thanks for the comments. I'll make those changes to the design document and send it out again. One question regarding something Phil brought up before. Should we consider moving the namelist config variables into a decomp group, rather than having them in IO? I previously left them IO just because there were only going to be a total of 3, so it didn't necessitate a new namelist group in my opinion. Though, at a later point in time we might add other namelist variables to this group, so it might make more sense then.<br>
<br>Thanks again.<br><br>Doug<br><br><div class="gmail_quote">On Mon, Jan 30, 2012 at 1:34 PM, Michael Duda <span dir="ltr"><<a href="mailto:duda@ucar.edu">duda@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, Doug.<br>
<br>
As Phil says, it should be easy to remove the local block ID in<br>
future if it turns out that we don't make good use of it.<br>
<br>
Regarding the block information returned by<br>
mpas_block_decomp_cells_for_proc(), I think it would make sense to<br>
just simplify the block information before it is returned by the<br>
routine, rather than having code that calls the routine perform<br>
the work. Other than this change, I think the design looks pretty<br>
good!<br>
<br>
Cheers,<br>
Michael<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Mon, Jan 30, 2012 at 11:23:01AM -0700, Doug Jacobsen wrote:<br>
> Hi Michael,<br>
><br>
> I was thinking a bit more about the design document this weekend, and, with<br>
> > the blocks stored in a linked-list, I'm wondering whether we really need<br>
> > local block indices. The main use of local indices that comes to my mind<br>
> > would be in indexing arrays of blocks, though I'm not sure whether we'll<br>
> > need to do that; perhaps we will optimize the shared-memory copies using<br>
> > arrays of block pointers? Adding a bit of extra code to deal with local<br>
> > indices seems straightforward enough and might be useful in future, but the<br>
> > overall design might be simpler if blocks were just identified by a unique,<br>
> > global ID.<br>
> ><br>
><br>
> I'm not sure how useful the local block id inside of the block_type would<br>
> be, I just figured it might be helpful to have. Especially if we had to<br>
> swap local blocks with id's 1 between two processors later. It might be<br>
> easier to determine which blocks we are talking about, especially when<br>
> their global block id's wouldn't directly map to those local block id's<br>
> anymore. Of course, for the short term, and the goals of this project we<br>
> would not be tackling that problem and it wouldn't really be of any use<br>
> yet. It was most of a long term goal that I was trying to prepare for, with<br>
> Phil's mention of dynamic load balancing.<br>
><br>
><br>
> > Also, would it improve usability of the mpas_block_decomp_cells_for_proc()<br>
> > interface if we returned the block ownership information (i.e.,<br>
> > local_block_list)<br>
> > in a different form? For example, it's not specified which order, if any,<br>
> > the cells in local_cell_list will be returned (alternatively, there's no<br>
> > guarantee that local_block_list will be in sorted order); so, perhaps we<br>
> > can always return cells that are in the same block contiguously in<br>
> > local_cell_list, and then provide block information with arrays<br>
> ><br>
> > block_id(:) - global IDs of the blocks<br>
> > NB: size(block_id) = number of blocks owned by the task<br>
> ><br>
> > block_start(:) - offset in local_cell_list of the first cell in the<br>
> > blocks<br>
> ><br>
> > block_count(:) - number of cells in the blocks<br>
> ><br>
> > The above is obviously just one alternative to local_block_list, but it<br>
> > might make it easier for the caller to quickly determine how many blocks it<br>
> > is being given, the sizes of those blocks, and the global cell IDs in each<br>
> > of those blocks.<br>
> ><br>
><br>
> So, in this case local_cell_list would remain the same, but it would be<br>
> sorted by local_block_id right? I could see this method being easier for<br>
> the IO module to handle, since it can easily build one block at a time<br>
> rather than having to loop over both local_cell_list and local_block_list<br>
> to determine which cells are in a specific block. It's really just a matter<br>
> of if we want to sort the structures inside of the<br>
> mpas_block_decomp_cells_for_proc call, or after the data is returned.<br>
> Either one is fine with me, so since you are tackling the IO rewrite if you<br>
> would rather push this into the block_decomp modules I can take care of<br>
> that.<br>
><br>
> Thanks for the comments,<br>
> Doug<br>
><br>
><br>
> On Fri, Jan 27, 2012 at 06:49:59PM -0700, Doug Jacobsen wrote:<br>
> > Thanks for the comments Phil,<br>
> ><br>
> > I fixed the typo you mentioned, and I have already changed the design<br>
> > document to only provide a public interface to the<br>
> > mpas_get_blocks_per_proc, rather than the lookup table.<br>
> ><br>
> > Considering dynamic load balancing, would it be useful to store the local<br>
> > block id, and owning processor number within the block derived data type?<br>
> I<br>
> > could see at least the local block id being useful, to help you if you<br>
> need<br>
> > to rearrange the block indices, or insert a new block into the correct<br>
> > position, but I am unsure about the owning proc information. I guess that<br>
> > would just be dminfo % my_proc_id, since you would only have blocks that<br>
> > you would own.<br>
> ><br>
> > If anyone has any input on these ideas please let me know, but for now<br>
> I'll<br>
> > add the local block index to the block data type (in the design doc), and<br>
> > leave out the owning proc id.<br>
> ><br>
> > Thanks again,<br>
> > Doug<br>
> ><br>
> > On Fri, Jan 27, 2012 at 3:52 PM, Jones, Philip W <<a href="mailto:pwjones@lanl.gov">pwjones@lanl.gov</a>> wrote:<br>
> ><br>
> > ><br>
> > > Doug,<br>
> > ><br>
><br>
> > > > Generally, I think it?s ok. Couple of minor things below. Michael is<br>
> > > > correct in saying we want to be careful about holding onto arrays that<br>
> > are<br>
> > > > proportional to either number of tasks/blocks, even if they appear<br>
> > small<br>
> > > > now. Memory per node is not growing and on a per-core basis will<br>
> > continue<br>
> > > > to shrink. I suspect we will only need these functions on<br>
> > > > start-up/initialization and may be able to deallocate after this phase<br>
> > is<br>
> > > > complete (and then the functions would return an error). Once we?ve<br>
> > set up<br>
> > > > communicators/patterns for halos, reductions and I/O, I don?t believe<br>
> > we?ll<br>
> > > > have a need for these sorts of global maps. Even if we did dynamic<br>
> > load<br>
> > > > balancing, we?d typically be passing off a block or two from one task<br>
> > to<br>
> > > > another and can update locally without these global queries. Maybe we<br>
> > can<br>
> > > > keep these around for now and worry about this later.<br>
> > > ><br>
> > > > For minor issues:<br>
> > > ><br>
> > > > Do we want this in the io namelist block? I think there should be a<br>
> > > > general decomposition group rather than putting this in i/o. (I have<br>
> > other<br>
> > > > naming preferences too, but that?ll open another can o? worms).<br>
> > > ><br>
> > > > The last para on page 6 of the pdf is missing something ?wher is the<br>
> > > > number of proc?. It appears in the tex input too, so it?s not a tex<br>
> > issue.<br>
> > > ><br>
> > > > Phil<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On 1/27/12 2:53 PM, "Doug Jacobsen" <<a href="mailto:jacobsen.douglas@gmail.com">jacobsen.douglas@gmail.com</a>><br>
> > wrote:<br>
> > > ><br>
> > > > Hi again all,<br>
> > > ><br>
> > > > I have now attached the newest version of this design doc.<br>
> > > ><br>
> > > > Please let me know of any other comments/questions.<br>
> > > ><br>
> > > > Doug<br>
> > > ><br>
> > > > ------------------------------<br>
> > > > _______________________________________________<br>
> > > > mpas-developers mailing list<br>
> > > > <a href="mailto:mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br>
> > > > <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers" target="_blank">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > ---<br>
> > > > Correspondence/TSPA/DUSA EARTH<br>
> > > > ------------------------------------------------------------<br>
> > > > Philip Jones <a href="mailto:pwjones@lanl.gov">pwjones@lanl.gov</a><br>
> > > > Climate, Ocean and Sea Ice Modeling<br>
> > > > Los Alamos National Laboratory<br>
> > > > T-3 MS B216 Ph: <a href="tel:505-500-2699" value="+15055002699">505-500-2699</a><br>
> > > > PO Box 1663 Fax: <a href="tel:505-665-5926" value="+15056655926">505-665-5926</a><br>
> > > > Los Alamos, NM 87545-1663<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> ><br>
> > > _______________________________________________<br>
> > > mpas-developers mailing list<br>
> > > <a href="mailto:mpas-developers@mailman.ucar.edu">mpas-developers@mailman.ucar.edu</a><br>
> > > <a href="http://mailman.ucar.edu/mailman/listinfo/mpas-developers" target="_blank">http://mailman.ucar.edu/mailman/listinfo/mpas-developers</a><br>
> ><br>
> ><br>
</div></div></blockquote></div><br>