Phil,<br><br>Yeah, I figured that max would give you the most relevant information to see where large time is spent on any processor. The min wouldn't be too difficult to print as well. I figured average would either be fairly representative of the total set of processors, or it would get corrupted by load imbalances if they were really bad.<br>
<br>The global reductions are done at the end of the simulation, and they are only called if the domain_info variable points to domain % dminfo. Which means that the mpas_core line I previously mentioned needs to be added to the initialization phase.<br>
<br>In terms of min and max, the global min and max times for a single call are still printed. And the max total time spent in a routine (for a processor) is printed. I could determine the min total time spent in a routine (for a processor) as well if that's something everyone is interested in. As a summary of the current information that is printed. Below is a line from the current timer output.<br>
<br> 1 initialize 6.56141 1 6.56141 6.56141 6.56141 0.01<br><br>The first column tells you how many timers were started prior to starting this one. The second tells you the timer's name. Third is the total time spent within this timer, including all starts and stops. Fourth is the number of times this timer was started and stopped. Fourth is the min time for a single call to this timer. Fifth is the max time for a single call to this timer. Sixth is the average time for a single call to this timer (or column 3 / colum 4) and finally is the percent of the total_time timer that was spent in this timer.<br>
<br>The global reduction takes the global max of columns 3 and 5, and the global min of colum 6. And columns 7 and 8 are recomputed using the new global max of column 3. <br><br>Let me know if there is any more information anyone wants in the timers.<br>
<br>Doug <br><br><div class="gmail_quote">On Fri, Dec 9, 2011 at 8:35 AM, Jones, Philip W <span dir="ltr"><<a href="mailto:pwjones@lanl.gov">pwjones@lanl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div><br>
</div>
Doug,
<div><br>
</div>
<div>Max time is most relevant and printing the min time is useful to get an idea of load imbalance. Don't think the mean really tells you much.</div>
<div><br>
</div>
<div>And the global reductions are only in the timer print, right?<br>
<div><br>
<div><font size="2">
<div>Thanks,<br>
<br>
Phil<br>
<br>
TSPA/Correspondence/DUSA EARTH<br>
------<br>
Philip Jones (<a href="mailto:pwjones@lanl.gov" target="_blank">pwjones@lanl.gov</a>)<br>
Climate, Ocean and Sea Ice Modeling<br>
Los Alamos National Laboratory<br>
T-3 MS B216<br>
P.O. Box 1663<br>
Los Alamos, NM 87545<br>
</div>
</font></div>
</div>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma" size="2"><b>From:</b> <a href="mailto:mpas-developers-bounces@mailman.ucar.edu" target="_blank">mpas-developers-bounces@mailman.ucar.edu</a> [<a href="mailto:mpas-developers-bounces@mailman.ucar.edu" target="_blank">mpas-developers-bounces@mailman.ucar.edu</a>] on behalf of Doug Jacobsen [<a href="mailto:jacobsen.douglas@gmail.com" target="_blank">jacobsen.douglas@gmail.com</a>]<br>
<b>Sent:</b> Thursday, December 08, 2011 7:36 PM<br>
<b>To:</b> <a href="mailto:mpas-developers@ucar.edu" target="_blank">mpas-developers@ucar.edu</a><br>
<b>Subject:</b> Re: [mpas-developers] mpas_timer.F synchronization issue.<br>
</font><br>
</div><div><div></div><div class="h5">
<div></div>
<div>Hi Everyone,<br>
<br>
Something else that I would like input on regarding this. I currently have two options for synchronizing the timers. First is the current version, which just uses the max of all of the processors timers. The other option would be to average all of the timers
across processors. Each have their own benefits and provide slightly different information. So if anyone has any preferences it would be good to have a discussion about them.<br>
<br>
Thanks!<br>
Doug<br>
<br>
<div class="gmail_quote">On Thu, Dec 8, 2011 at 3:47 PM, Doug Jacobsen <span dir="ltr">
<<a href="mailto:jacobsen.douglas@gmail.com" target="_blank">jacobsen.douglas@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello Everyone,<br>
<br>
I recently noticed that when running an MPI job processors would report different times for sub-timers, ie. not including the total_time timer. This is mostly due to some processors having to wait for mpi calls to finish while other ones don't. None of the
previous versions of mpas_timer.F have supported making sure the timers report the same time over all of the processors. So I have attached a new version of mpas_timer.F that supports this. It essentially makes each timer's total time the maximum total time
over all of the processors. It also gets the global max and min single call time to print as well. I think this gives a better over all profile for the time spent in routines rather than having to go through each processors log.*.out file to see how it behaved.<br>
<br>
To support this, the timer module now stores a pointer to domain % dminfo so you don't have to pass it in to print out the timers. Doing this allows the current timer implementation to stay the same, and allows the syncing of timers by adding a single line
to mpas_*_mpas_core.F within each core, which is:<br>
<br>
call mpas_timer_init(domain)<br>
<br>
within mpas_core_init.<br>
<br>
I'm open to any comments or suggestions regarding this change, but I would like to propagate it to the trunk. I will also propagate the above addition to mpas_ocn_mpas_core.F but can add it to the other cores if requested.
<br>
<br>
Thanks for your input.<br>
<font color="#888888">Doug<br>
</font></blockquote>
</div>
<br>
</div>
</div></div></div>
</div>
</div>
</div>
</blockquote></div><br>