[mpas-developers] Block Decomposition Revision
Jones, Philip W
pwjones at lanl.gov
Mon Jan 30 12:52:22 MST 2012
Guys,
I guess the local block ID was useful to us in previous implementations both for packing/unpacking messages and for indexing into arrays since we had an array block index rather than a linked list. But you can certainly do this with only a global id, the local index just saves a search step. In the end, I think this is a relatively simple change that we can either add/remove later.
Phil
On 1/30/12 11:23 AM, "Doug Jacobsen" <jacobsen.douglas at gmail.com> wrote:
Hi Michael,
I was thinking a bit more about the design document this weekend, and, with
the blocks stored in a linked-list, I'm wondering whether we really need
local block indices. The main use of local indices that comes to my mind
would be in indexing arrays of blocks, though I'm not sure whether we'll
need to do that; perhaps we will optimize the shared-memory copies using
arrays of block pointers? Adding a bit of extra code to deal with local
indices seems straightforward enough and might be useful in future, but the
overall design might be simpler if blocks were just identified by a unique,
global ID.
I'm not sure how useful the local block id inside of the block_type would be, I just figured it might be helpful to have. Especially if we had to swap local blocks with id's 1 between two processors later. It might be easier to determine which blocks we are talking about, especially when their global block id's wouldn't directly map to those local block id's anymore. Of course, for the short term, and the goals of this project we would not be tackling that problem and it wouldn't really be of any use yet. It was most of a long term goal that I was trying to prepare for, with Phil's mention of dynamic load balancing.
Also, would it improve usability of the mpas_block_decomp_cells_for_proc()
interface if we returned the block ownership information (i.e., local_block_list)
in a different form? For example, it's not specified which order, if any,
the cells in local_cell_list will be returned (alternatively, there's no
guarantee that local_block_list will be in sorted order); so, perhaps we
can always return cells that are in the same block contiguously in
local_cell_list, and then provide block information with arrays
block_id(:) - global IDs of the blocks
NB: size(block_id) = number of blocks owned by the task
block_start(:) - offset in local_cell_list of the first cell in the blocks
block_count(:) - number of cells in the blocks
The above is obviously just one alternative to local_block_list, but it
might make it easier for the caller to quickly determine how many blocks it
is being given, the sizes of those blocks, and the global cell IDs in each
of those blocks.
So, in this case local_cell_list would remain the same, but it would be sorted by local_block_id right? I could see this method being easier for the IO module to handle, since it can easily build one block at a time rather than having to loop over both local_cell_list and local_block_list to determine which cells are in a specific block. It's really just a matter of if we want to sort the structures inside of the mpas_block_decomp_cells_for_proc call, or after the data is returned. Either one is fine with me, so since you are tackling the IO rewrite if you would rather push this into the block_decomp modules I can take care of that.
Thanks for the comments,
Doug
On Fri, Jan 27, 2012 at 06:49:59PM -0700, Doug Jacobsen wrote:
> Thanks for the comments Phil,
>
> I fixed the typo you mentioned, and I have already changed the design
> document to only provide a public interface to the
> mpas_get_blocks_per_proc, rather than the lookup table.
>
> Considering dynamic load balancing, would it be useful to store the local
> block id, and owning processor number within the block derived data type? I
> could see at least the local block id being useful, to help you if you need
> to rearrange the block indices, or insert a new block into the correct
> position, but I am unsure about the owning proc information. I guess that
> would just be dminfo % my_proc_id, since you would only have blocks that
> you would own.
>
> If anyone has any input on these ideas please let me know, but for now I'll
> add the local block index to the block data type (in the design doc), and
> leave out the owning proc id.
>
> Thanks again,
> Doug
>
> On Fri, Jan 27, 2012 at 3:52 PM, Jones, Philip W <pwjones at lanl.gov> wrote:
>
> >
> > Doug,
> >
> > Generally, I think it?s ok. Couple of minor things below. Michael is
> > correct in saying we want to be careful about holding onto arrays that are
> > proportional to either number of tasks/blocks, even if they appear small
> > now. Memory per node is not growing and on a per-core basis will continue
> > to shrink. I suspect we will only need these functions on
> > start-up/initialization and may be able to deallocate after this phase is
> > complete (and then the functions would return an error). Once we?ve set up
> > communicators/patterns for halos, reductions and I/O, I don?t believe we?ll
> > have a need for these sorts of global maps. Even if we did dynamic load
> > balancing, we?d typically be passing off a block or two from one task to
> > another and can update locally without these global queries. Maybe we can
> > keep these around for now and worry about this later.
> >
> > For minor issues:
> >
> > Do we want this in the io namelist block? I think there should be a
> > general decomposition group rather than putting this in i/o. (I have other
> > naming preferences too, but that?ll open another can o? worms).
> >
> > The last para on page 6 of the pdf is missing something ?wher is the
> > number of proc?. It appears in the tex input too, so it?s not a tex issue.
> >
> > Phil
> >
> >
> >
> > On 1/27/12 2:53 PM, "Doug Jacobsen" <jacobsen.douglas at gmail.com> wrote:
> >
> > Hi again all,
> >
> > I have now attached the newest version of this design doc.
> >
> > Please let me know of any other comments/questions.
> >
> > Doug
> >
> > ------------------------------
> > _______________________________________________
> > mpas-developers mailing list
> > mpas-developers at mailman.ucar.edu
> > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> >
> >
> >
> > ---
> > Correspondence/TSPA/DUSA EARTH
> > ------------------------------------------------------------
> > Philip Jones pwjones at lanl.gov
> > Climate, Ocean and Sea Ice Modeling
> > Los Alamos National Laboratory
> > T-3 MS B216 Ph: 505-500-2699 <tel:505-500-2699>
> > PO Box 1663 Fax: 505-665-5926 <tel:505-665-5926>
> > Los Alamos, NM 87545-1663
> >
> >
> >
> >
> _______________________________________________
> mpas-developers mailing list
> mpas-developers at mailman.ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/mpas-developers
________________________________
_______________________________________________
mpas-developers mailing list
mpas-developers at mailman.ucar.edu
http://mailman.ucar.edu/mailman/listinfo/mpas-developers
---
Correspondence/TSPA/DUSA EARTH
------------------------------------------------------------
Philip Jones pwjones at lanl.gov
Climate, Ocean and Sea Ice Modeling
Los Alamos National Laboratory
T-3 MS B216 Ph: 505-500-2699
PO Box 1663 Fax: 505-665-5926
Los Alamos, NM 87545-1663
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120130/7fa2a1ef/attachment.html
More information about the mpas-developers
mailing list