<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Lucida Handwriting";
panose-1:3 1 1 1 1 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Brian,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The wrf domain is too small to do such performance test.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Use a larger, much large domain, you’ll a better performance number.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="background:white"><b><span style="font-size:17.0pt;font-family:"Lucida Handwriting";color:#1F497D">Wei Huang</span></b><span style="font-size:9.5pt;font-family:"Lucida Handwriting";color:#222222"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> wrf-users-bounces@ucar.edu [mailto:wrf-users-bounces@ucar.edu]
<b>On Behalf Of </b>Andrus, Brian Contractor<br>
<b>Sent:</b> Friday, July 24, 2015 1:52 PM<br>
<b>To:</b> wrf-users@ucar.edu<br>
<b>Subject:</b> [Wrf-users] Optimizing OMP_NUM_THREADS<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am a little confused about running wrf. I have built it and am following the testing for the Jan 2000 data set per:
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www2.mmm.ucar.edu_wrf_OnLineTutorial_CASES_JAN00_wrf.htm&d=BQMFAg&c=qHNyRuJKHYeI-vwTnTfWXq4fkZpyjWUA1LcPL7eQSSQ&r=6RmpIe4P-G8omsHqYS85uBS-3JaNVk3lvOG-hmdOrr8&m=beCRANFC6K_f6XDzLxHv3fxVEyzYKzTanLPeE9nlkY8&s=Nr6hVplJ77yATKxwTJ4fkRR1GIMQ9BvlgSOeUL1AEFY&e=">
http://www2.mmm.ucar.edu/wrf/OnLineTutorial/CASES/JAN00/wrf.htm</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Everything does work and I can get proper output, however I have noticed unusual timing that I cannot figure why.<o:p></o:p></p>
<p class="MsoNormal">I am running on a system with 64 cores and 256GB RAM.<o:p></o:p></p>
<p class="MsoNormal">I compiled with <o:p></o:p></p>
<p class="MsoNormal"> 1) compile/pgi/15.7 2) mpi/openmpi/1.8.5 3) app/netcdf/4.3.3.1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Options were for dm+sm (option 55 pgf90/pgcc) and basic nesting (option 1)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Now I am running wrf and producing wrfout_d01_2000-01-24_12:00:00<o:p></o:p></p>
<p class="MsoNormal">What is odd is the extreme variation with different OMP_NUM_THREADS set.<o:p></o:p></p>
<p class="MsoNormal">It seems it is best at 4. Any more or less and the time it takes increases.<o:p></o:p></p>
<p class="MsoNormal">Setting to 8 is close to the same as setting it to 2<o:p></o:p></p>
<p class="MsoNormal">Setting it to 64 and it takes almost 4 times as long as 4..??<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here are some timings:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ export OMP_NUM_THREADS=1<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ time ./wrf.exe<o:p></o:p></i></p>
<p class="MsoNormal"><i>starting wrf task 0 of 1<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal"><i>real 2m51.743s<o:p></o:p></i></p>
<p class="MsoNormal"><i>user 2m39.087s<o:p></o:p></i></p>
<p class="MsoNormal"><i>sys 0m12.277s<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ export OMP_NUM_THREADS=2<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ time ./wrf.exe<o:p></o:p></i></p>
<p class="MsoNormal"><i>starting wrf task 0 of 1<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal"><i>real 1m49.172s<o:p></o:p></i></p>
<p class="MsoNormal"><i>user 3m15.582s<o:p></o:p></i></p>
<p class="MsoNormal"><i>sys 0m19.015s<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ export OMP_NUM_THREADS=4<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ time ./wrf.exe<o:p></o:p></i></p>
<p class="MsoNormal"><i>starting wrf task 0 of 1<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal"><i>real 1m27.357s<o:p></o:p></i></p>
<p class="MsoNormal"><i>user 4m42.111s<o:p></o:p></i></p>
<p class="MsoNormal"><i>sys 0m35.187s<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ export OMP_NUM_THREADS=8<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ time ./wrf.exe<o:p></o:p></i></p>
<p class="MsoNormal"><i>starting wrf task 0 of 1<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal"><i>real 1m35.480s<o:p></o:p></i></p>
<p class="MsoNormal"><i>user 8m20.966s<o:p></o:p></i></p>
<p class="MsoNormal"><i>sys 1m13.376s<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ export OMP_NUM_THREADS=16<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ time ./wrf.exe<o:p></o:p></i></p>
<p class="MsoNormal"><i>starting wrf task 0 of 1<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal"><i>real 1m52.862s<o:p></o:p></i></p>
<p class="MsoNormal"><i>user 15m43.787s<o:p></o:p></i></p>
<p class="MsoNormal"><i>sys 2m4.978s<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ export OMP_NUM_THREADS=64
<o:p></o:p></i></p>
<p class="MsoNormal"><i>[bdandrus@compute-7-3 em_real]$ time ./wrf.exe<o:p></o:p></i></p>
<p class="MsoNormal"><i>starting wrf task 0 of 1<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal"><i>real 5m54.857s<o:p></o:p></i></p>
<p class="MsoNormal"><i>user 197m37.807s<o:p></o:p></i></p>
<p class="MsoNormal"><i>sys 7m57.993s<o:p></o:p></i></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any ideas as to what would cause that?<o:p></o:p></p>
<p class="MsoNormal">I have all but given up on using mpirun as that seems to make it take HOURS no matter how many procs/threads I set. I do see it running 100%cpu on each core it is assigned when I do that, but it rarely writes anything.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brian Andrus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">ITACS/Research Computing<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Naval Postgraduate School<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Monterey, California<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">voice: 831-656-6238<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>