[mpas-developers] Block Decomposition Revision
Doug Jacobsen
jacobsen.douglas at gmail.com
Mon Jan 30 14:10:45 MST 2012
Alright everyone,
Here is another update of the design document. It should incorporate all of
the changes, and it's available in the repository too. Again let me know if
there are any comments/questions.
Thanks,
Doug
On Mon, Jan 30, 2012 at 1:59 PM, Michael Duda <duda at ucar.edu> wrote:
> I'd agree: a separate namelist record would be good.
>
> Michael
>
>
> On Mon, Jan 30, 2012 at 01:54:08PM -0700, Todd Ringler wrote:
> >
> > For clarity and possible extensions in the future, I think that a
> separate namelist header for decomposition would be appropriate.
> >
> > On Jan 30, 2012, at 1:41 PM, Doug Jacobsen wrote:
> >
> > > Hi Michael and Phil,
> > >
> > > Thanks for the comments. I'll make those changes to the design
> document and send it out again. One question regarding something Phil
> brought up before. Should we consider moving the namelist config variables
> into a decomp group, rather than having them in IO? I previously left them
> IO just because there were only going to be a total of 3, so it didn't
> necessitate a new namelist group in my opinion. Though, at a later point in
> time we might add other namelist variables to this group, so it might make
> more sense then.
> > >
> > > Thanks again.
> > >
> > > Doug
> > >
> > > On Mon, Jan 30, 2012 at 1:34 PM, Michael Duda <duda at ucar.edu> wrote:
> > > Hi, Doug.
> > >
> > > As Phil says, it should be easy to remove the local block ID in
> > > future if it turns out that we don't make good use of it.
> > >
> > > Regarding the block information returned by
> > > mpas_block_decomp_cells_for_proc(), I think it would make sense to
> > > just simplify the block information before it is returned by the
> > > routine, rather than having code that calls the routine perform
> > > the work. Other than this change, I think the design looks pretty
> > > good!
> > >
> > > Cheers,
> > > Michael
> > >
> > >
> > > On Mon, Jan 30, 2012 at 11:23:01AM -0700, Doug Jacobsen wrote:
> > > > Hi Michael,
> > > >
> > > > I was thinking a bit more about the design document this weekend,
> and, with
> > > > > the blocks stored in a linked-list, I'm wondering whether we
> really need
> > > > > local block indices. The main use of local indices that comes to
> my mind
> > > > > would be in indexing arrays of blocks, though I'm not sure whether
> we'll
> > > > > need to do that; perhaps we will optimize the shared-memory copies
> using
> > > > > arrays of block pointers? Adding a bit of extra code to deal with
> local
> > > > > indices seems straightforward enough and might be useful in
> future, but the
> > > > > overall design might be simpler if blocks were just identified by
> a unique,
> > > > > global ID.
> > > > >
> > > >
> > > > I'm not sure how useful the local block id inside of the block_type
> would
> > > > be, I just figured it might be helpful to have. Especially if we had
> to
> > > > swap local blocks with id's 1 between two processors later. It might
> be
> > > > easier to determine which blocks we are talking about, especially
> when
> > > > their global block id's wouldn't directly map to those local block
> id's
> > > > anymore. Of course, for the short term, and the goals of this
> project we
> > > > would not be tackling that problem and it wouldn't really be of any
> use
> > > > yet. It was most of a long term goal that I was trying to prepare
> for, with
> > > > Phil's mention of dynamic load balancing.
> > > >
> > > >
> > > > > Also, would it improve usability of the
> mpas_block_decomp_cells_for_proc()
> > > > > interface if we returned the block ownership information (i.e.,
> > > > > local_block_list)
> > > > > in a different form? For example, it's not specified which order,
> if any,
> > > > > the cells in local_cell_list will be returned (alternatively,
> there's no
> > > > > guarantee that local_block_list will be in sorted order); so,
> perhaps we
> > > > > can always return cells that are in the same block contiguously in
> > > > > local_cell_list, and then provide block information with arrays
> > > > >
> > > > > block_id(:) - global IDs of the blocks
> > > > > NB: size(block_id) = number of blocks owned by
> the task
> > > > >
> > > > > block_start(:) - offset in local_cell_list of the first cell in
> the
> > > > > blocks
> > > > >
> > > > > block_count(:) - number of cells in the blocks
> > > > >
> > > > > The above is obviously just one alternative to local_block_list,
> but it
> > > > > might make it easier for the caller to quickly determine how many
> blocks it
> > > > > is being given, the sizes of those blocks, and the global cell IDs
> in each
> > > > > of those blocks.
> > > > >
> > > >
> > > > So, in this case local_cell_list would remain the same, but it would
> be
> > > > sorted by local_block_id right? I could see this method being easier
> for
> > > > the IO module to handle, since it can easily build one block at a
> time
> > > > rather than having to loop over both local_cell_list and
> local_block_list
> > > > to determine which cells are in a specific block. It's really just a
> matter
> > > > of if we want to sort the structures inside of the
> > > > mpas_block_decomp_cells_for_proc call, or after the data is returned.
> > > > Either one is fine with me, so since you are tackling the IO rewrite
> if you
> > > > would rather push this into the block_decomp modules I can take care
> of
> > > > that.
> > > >
> > > > Thanks for the comments,
> > > > Doug
> > > >
> > > >
> > > > On Fri, Jan 27, 2012 at 06:49:59PM -0700, Doug Jacobsen wrote:
> > > > > Thanks for the comments Phil,
> > > > >
> > > > > I fixed the typo you mentioned, and I have already changed the
> design
> > > > > document to only provide a public interface to the
> > > > > mpas_get_blocks_per_proc, rather than the lookup table.
> > > > >
> > > > > Considering dynamic load balancing, would it be useful to store
> the local
> > > > > block id, and owning processor number within the block derived
> data type?
> > > > I
> > > > > could see at least the local block id being useful, to help you if
> you
> > > > need
> > > > > to rearrange the block indices, or insert a new block into the
> correct
> > > > > position, but I am unsure about the owning proc information. I
> guess that
> > > > > would just be dminfo % my_proc_id, since you would only have
> blocks that
> > > > > you would own.
> > > > >
> > > > > If anyone has any input on these ideas please let me know, but for
> now
> > > > I'll
> > > > > add the local block index to the block data type (in the design
> doc), and
> > > > > leave out the owning proc id.
> > > > >
> > > > > Thanks again,
> > > > > Doug
> > > > >
> > > > > On Fri, Jan 27, 2012 at 3:52 PM, Jones, Philip W <pwjones at lanl.gov>
> wrote:
> > > > >
> > > > > >
> > > > > > Doug,
> > > > > >
> > > >
> > > > > > > Generally, I think it?s ok. Couple of minor things below.
> Michael is
> > > > > > > correct in saying we want to be careful about holding onto
> arrays that
> > > > > are
> > > > > > > proportional to either number of tasks/blocks, even if they
> appear
> > > > > small
> > > > > > > now. Memory per node is not growing and on a per-core basis
> will
> > > > > continue
> > > > > > > to shrink. I suspect we will only need these functions on
> > > > > > > start-up/initialization and may be able to deallocate after
> this phase
> > > > > is
> > > > > > > complete (and then the functions would return an error). Once
> we?ve
> > > > > set up
> > > > > > > communicators/patterns for halos, reductions and I/O, I don?t
> believe
> > > > > we?ll
> > > > > > > have a need for these sorts of global maps. Even if we did
> dynamic
> > > > > load
> > > > > > > balancing, we?d typically be passing off a block or two from
> one task
> > > > > to
> > > > > > > another and can update locally without these global queries.
> Maybe we
> > > > > can
> > > > > > > keep these around for now and worry about this later.
> > > > > > >
> > > > > > > For minor issues:
> > > > > > >
> > > > > > > Do we want this in the io namelist block? I think there
> should be a
> > > > > > > general decomposition group rather than putting this in i/o.
> (I have
> > > > > other
> > > > > > > naming preferences too, but that?ll open another can o? worms).
> > > > > > >
> > > > > > > The last para on page 6 of the pdf is missing something ?wher
> is the
> > > > > > > number of proc?. It appears in the tex input too, so it?s not
> a tex
> > > > > issue.
> > > > > > >
> > > > > > > Phil
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 1/27/12 2:53 PM, "Doug Jacobsen" <
> jacobsen.douglas at gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Hi again all,
> > > > > > >
> > > > > > > I have now attached the newest version of this design doc.
> > > > > > >
> > > > > > > Please let me know of any other comments/questions.
> > > > > > >
> > > > > > > Doug
> > > > > > >
> > > > > > > ------------------------------
> > > > > > > _______________________________________________
> > > > > > > mpas-developers mailing list
> > > > > > > mpas-developers at mailman.ucar.edu
> > > > > > > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ---
> > > > > > > Correspondence/TSPA/DUSA EARTH
> > > > > > > ------------------------------------------------------------
> > > > > > > Philip Jones pwjones at lanl.gov
> > > > > > > Climate, Ocean and Sea Ice Modeling
> > > > > > > Los Alamos National Laboratory
> > > > > > > T-3 MS B216 Ph: 505-500-2699
> > > > > > > PO Box 1663 Fax: 505-665-5926
> > > > > > > Los Alamos, NM 87545-1663
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > > > _______________________________________________
> > > > > > mpas-developers mailing list
> > > > > > mpas-developers at mailman.ucar.edu
> > > > > > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> > > > >
> > > > >
> > >
> > > _______________________________________________
> > > mpas-developers mailing list
> > > mpas-developers at mailman.ucar.edu
> > > http://mailman.ucar.edu/mailman/listinfo/mpas-developers
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120130/58aa5a85/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpas_block_decomp_redesign.pdf
Type: application/pdf
Size: 123768 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120130/58aa5a85/attachment-0001.pdf
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpas_block_decomp_redesign.tex
Type: application/x-tex
Size: 14647 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/mpas-developers/attachments/20120130/58aa5a85/attachment-0001.tex
More information about the mpas-developers
mailing list