<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Yaoping,<div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div id="divtagdefaultwrapper" class="" style="font-size: 10pt; background-color: rgb(255, 255, 255); font-family: Mangal, sans-serif;"><div class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="" style="font-size: 10pt;">The segfault problem with OpenMP seems hopeless. The "ulimit -s unlimited" command did not help. I also debugged another error, but it did not help either. I am giving it up for now. </span></div></div></blockquote></div><div class=""><div class="" style="font-size: 10pt; background-color: rgb(255, 255, 255); font-family: Mangal, sans-serif;"><div class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="" style="font-size: 10pt;"><br class=""></span></div></div></div><div class="">did you try to set/increase the OMP stacksize? You can set the limit using the environment variable OMP_STACKSIZE (KMP_STACKSIZE for Intel compilers).</div><div class=""><br class=""></div><div class="">Cheers</div><div class=""><br class=""></div><div class="">Dom</div><div class=""><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 3/09/2016, at 7:26 AM, Wang, Yaoping <<a href="mailto:wang.3866@buckeyemail.osu.edu" class="">wang.3866@buckeyemail.osu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div id="divtagdefaultwrapper" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 10pt; background-color: rgb(255, 255, 255); font-family: Mangal, sans-serif;" class=""><div style="margin-top: 0px; margin-bottom: 0px;" class="">Hi, </div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 13.3333px;" class="">This is embarrassing - </span><span style="font-size: 10pt;" class="">the speed problem was solved</span><span style="font-size: 10pt;" class="">.<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">The -d option was on when I built WRF months ago!</span></div><p style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""></span></p><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""><br class=""></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class="">For completeness, the compile-time error when building with openmp was: </span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""><br class=""></span></div><p style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""></span></p><div class="">../../frame/module_wrf_error.o: In function `MODULE_WRF_ERROR':</div><div class="">/fs/project/PAS0661/WRF3.8_3/WRFV3/frame/module_wrf_error.f90:4: undefined reference to `__kmpc_global_thread_num'</div><div class="">../../frame/module_wrf_error.o: In function `wrf_message':</div><div class="">/fs/project/PAS0661/WRF3.8_3/WRFV3/frame/module_wrf_error.f90:166: undefined reference to `__kmpc_global_thread_num'</div><div class="">/fs/project/PAS0661/WRF3.8_3/WRFV3/frame/module_wrf_error.f90:174: undefined reference to `__kmpc_master'</div><div class="">/fs/project/PAS0661/WRF3.8_3/WRFV3/frame/module_wrf_error.f90:174: undefined reference to `__kmpc_end_master'</div><div class="">../../frame/module_wrf_error.o: In function `wrf_message2':</div><div class="">/fs/project/PAS0661/WRF3.8_3/WRFV3/frame/module_wrf_error.f90:192: undefined reference to `__kmpc_global_thread_num'</div><div class="">/fs/project/PAS0661/WRF3.8_3/WRFV3/frame/module_wrf_error.f90:195: undefined reference to `__kmpc_master'</div><div class="">/fs/project/PAS0661/WRF3.8_3/WRFV3/frame/module_wrf_error.f90:195: undefined reference to `__kmpc_end_master'</div><div class="">make[2]: [diffwrf] Error 1 (ignored)</div><div class=""><br class=""></div>And was solved by adding "-openmp" into <span class="">FCBASEOPTS_NO_G</span><p style="margin-top: 0px; margin-bottom: 0px;" class=""></p><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Thank you,</div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Yaoping Wang</div></div><hr tabindex="-1" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline-block; width: 810.453125px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Reginaldo Ventura de Sá <<a href="mailto:regis@lamma.ufrj.br" class="">regis@lamma.ufrj.br</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Saturday, September 3, 2016 8:47:00 AM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Wang, Yaoping<br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [Wrf-users] What is a reasonable speed for WRF / how to increase it?</font><div class=""> </div></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div dir="ltr" class=""><div class=""><div class=""><div class=""><div class=""><br class=""></div>hello,<span class="Apple-converted-space"> </span><br class=""><br class=""></div>try the command:<br class=""><br class=""></div>ulimit -s unlimited <span class="Apple-converted-space"> </span><br class=""><br class=""><br class=""></div>Regis<br class=""><div class=""><br class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Sep 2, 2016 at 7:33 PM, Wang, Yaoping<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:wang.3866@buckeyemail.osu.edu" target="_blank" class="">wang.3866@buckeyemail.osu.edu</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div bgcolor="#FFFFFF" text="#000000" class=""><div style="font-size: 10pt; background-color: rgb(255, 255, 255); font-family: Mangal, sans-serif;" class=""><div style="margin-top: 0px; margin-bottom: 0px;" class="">Hi, </div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span class="">Could you explain more <span class="">about the "using 2 process in one core" and how to find out/address it? I am not very familiar with the technical aspect of supercomputing. I thought one core in supercomputing means one CPU, but did you mean that each CPU is further made up of multiple "little-cores" themselves, or 2 cores can be on the same CPU?</span></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span class=""><span class="">I am compiling MPI using the intel compiler</span></span><span style="font-size: 10pt;" class="">. But here is another problem. OpenMP</span><span style="font-size: 10pt;" class=""> does not work at all on my system. The compiler finishes successfully, but then, when I run "./real.exe", it segfaults without even creating an rsl file. I tried using "./configure -d", and updating my WRF version from 3.8 to 3.8.1, </span><span style="font-size: 10pt;" class="">but the segfault was the same.<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">Do you<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">know</span><span style="font-size: 10pt;" class=""><span class="Apple-converted-space"> </span></span><span style="font-size: 10pt;" class="">what else</span><span style="font-size: 10pt;" class=""> I might </span><span style="font-size: 10pt;" class="">do</span><span style="font-size: 10pt;" class="">? I attached my "configure.wrf" and "configure.wps" files. </span></div><p style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""></span></p><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span class=""><span class="">I also tested 44 cores with no luck. Increasing the number of nodes from 4 to 6 only increased the throughput marginally. </span></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span class=""><span class=""><br class=""></span></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span class=""><span class="">Thank you, </span></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span class=""><span class="">Yaoping</span></span></div></div><hr style="display: inline-block; width: 798.421875px;" class=""><div dir="ltr" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span><a href="mailto:wrf-users-bounces@ucar.edu" target="_blank" class="">wrf-users-bounces@ucar.edu</a><span class="Apple-converted-space"> </span><<a href="mailto:wrf-users-bounces@ucar.edu" target="_blank" class="">wrf-users-bounces@ucar.edu</a>> on behalf of Mike Dvorak <<a href="mailto:mike@sailtactics.com" target="_blank" class="">mike@sailtactics.com</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Thursday, September 1, 2016 5:05:28 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:wrf-users@ucar.edu" target="_blank" class="">wrf-users@ucar.edu</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [Wrf-users] What is a reasonable speed for WRF / how to increase it?</font><div class=""> </div></div><div class=""><div class="h5"><div class="">Hi Yaoping,<br class=""><br class="">What parallelization option did you compile WRF with (e.g. MPI only)? Also, I've found the Intel compilers to be 3 times faster than the GNU compilers on some WRF configurations (unfortunately). What compiler did you use?<br class=""><br class="">You may also want to experiment using less than the number of total cores on the machine. For example, you could try using 44 cores instead of 48. I think WRF EMS is set to do this by default. I've verified on some of my multi-core machines that this does indeed reduce the runtime.<br class=""><br class="">Cheers,<br class="">Mike<br class=""><br class=""><br class=""><div class="">On 09/01/2016 03:15 PM, Carlos Ross wrote:<br class=""></div><blockquote type="cite" class=""><div dir="ltr" class="">I think it should be faster, <span style="font-family: mangal, sans-serif; font-size: 13.3333px;" class="">Xeon x5650 CPUs are 6 cores and 12 threads, so you maybe using 2 process in one core and that is slowing it down.</span></div><div class="gmail_extra"><br class=""><div class="gmail_quote">2016-08-31 18:36 GMT-03:00 Wang, Yaoping<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:wang.3866@buckeyemail.osu.edu" target="_blank" class="">wang.3866@buckeyemail.osu.edu</a><wbr class="">></span>:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr" class=""><div style="font-size: 10pt; background-color: rgb(255, 255, 255); font-family: Mangal, sans-serif;" class=""><div style="margin-top: 0px; margin-bottom: 0px;" class="">Hi All, </div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">I am running WRF on a ~6km resolution, <span style="font-size: 10pt;" class="">91 x 121 domain in the eastern United States. I am using an adaptive time step which makes it mostly 72 sec increments. </span><span style="font-size: 10pt;" class="">There are 34 vertical levels. I use 4 x 12 cores on a Intel Xeon x5650 CPUs machine. </span><span style="font-size: 10pt;" class="">The throughput is about 1.2 hour wall time per 24 hours model time. </span></div><p style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""></span></p><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""><br class=""></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class="">Is this a reasonable speed? I found some information here (<a href="http://www.ecmwf.int/sites/default/files/elibrary/2014/13662-performance-analysis-operational-implementaion-wrf.pdf" target="_blank" class="">http://www.ecmwf.int/sites/de<wbr class="">fault/files/elibrary/2014/1366<wbr class="">2-performance-analysis-operati<wbr class="">onal-implementaion-wrf.pdf</a>) and after considering the domain difference, my run still seems a touch slow. And i</span><span style="font-size: 10pt;" class="">s there anyway I could figure how to make the model run faster?</span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class=""><br class=""></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class="">Thank you, </span></div><span class=""><font color="#888888" class=""><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 10pt;" class="">Yaoping Wang</span></div></font></span></div></div><br class="">______________________________<wbr class="">_________________<br class="">Wrf-users mailing list<br class=""><a href="mailto:Wrf-users@ucar.edu" target="_blank" class="">Wrf-users@ucar.edu</a><br class=""><a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" rel="noreferrer" target="_blank" class="">http://mailman.ucar.edu/mailma<wbr class="">n/listinfo/wrf-users</a><br class=""><br class=""></blockquote></div><br class=""></div><br class=""><fieldset class=""></fieldset><br class=""><pre class="">______________________________<wbr class="">_________________
Wrf-users mailing list
<a href="mailto:Wrf-users@ucar.edu" target="_blank" class="">Wrf-users@ucar.edu</a>
<a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" target="_blank" class="">http://mailman.ucar.edu/<wbr class="">mailman/listinfo/wrf-users</a>
</pre></blockquote><br class=""><div class="">--<span class="Apple-converted-space"> </span><br class=""><table height="126" border="0" cellpadding="2" cellspacing="2" width="15%" class=""><tbody class=""><tr class=""><td valign="center" class=""><span id="cid:part5.FD4A0143.10B55550@sailtactics.com"><logo.png></span><br class=""></td><td valign="top" class="">Mike Dvorak, PhD<br class="">Founder<br class="">Sail Tactics, LLC<br class="">Corpus Christi, TX<br class=""><a href="tel:%2B1%20650-454-5243" value="+16504545243" target="_blank" class="">+1 650-454-5243</a><br class=""><a href="http://sailtactics.com/" target="_blank" class="">http://sailtactics.com</a><br class=""></td></tr></tbody></table></div></div></div></div></div><br class="">______________________________<wbr class="">_________________<br class="">Wrf-users mailing list<br class=""><a href="mailto:Wrf-users@ucar.edu" class="">Wrf-users@ucar.edu</a><br class=""><a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" rel="noreferrer" target="_blank" class="">http://mailman.ucar.edu/<wbr class="">mailman/listinfo/wrf-users</a><br class=""><br class=""></blockquote></div><br class=""></div></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Wrf-users mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:Wrf-users@ucar.edu" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Wrf-users@ucar.edu</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://mailman.ucar.edu/mailman/listinfo/wrf-users</a></div></blockquote></div><br class=""></div></div></body></html>