Hi All<div><br></div><div>Thanks for everyone's advise. As I mentioned in last letter, Double domain don't work. So only single domain was tested. Thanks for Alex's advise. I will try it after enough computing resource given to me again. Here I want to share the test results using thousands of cores with WRF. </div>
<div><br></div><div>core used Integrating Time used</div><div>224 65min</div><div>1120 35min</div><div>2240 51min </div><div><br></div><div>The results show increasing cores didn't result to the expecting acceleration. Just like Don's advise, maybe the domain has been <span style>over-decomposed.</span></div>
<div><br></div><div><span style>Best Regards,</span>
</div><div><span style><br></span></div><div>brick</div><div><br><br><div class="gmail_quote">On Tue, Mar 27, 2012 at 6:00 AM, Alex Fierro <span dir="ltr"><<a href="mailto:alexandre.o.fierro@gmail.com">alexandre.o.fierro@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Greetings:<br><br>I have ran WRF on the Oak Ridge Supercomputers (jaguarpf XT-5) on 2000 cores 2 years ago and ran into a similar problem, which in my case was related to I/O quilting. <br>
<br>I had to select:<br><br> &namelist_quilt <br>
nio_tasks_per_group = 2, <br> nio_groups = 1, <br><br>and then everything went fine (for that particular case).<br><br>The simulations was ran for a 24-h period on a 4-km convective permitting grid over CONUS with a grid size of 1200 x 800 x 35. The simulation scaled well up to 8000 cores where, again, I/O caused some issues. In the past, I also believe Netcdf files had a hard-wired file size limit up to 2Gb (?) similar to Vis5d files. Have you tried also using native (raw) binaries for the output format?<br>
<br>Cheers and hope this helps,<br><br>Alexandre-<br>-- <br>-------------------------------------------------------------<br>Alexandre Fierro, PhD<br>Research Scientist-<br>National Severe Storms Laboratory (NSSL/NOAA)<br>
<b style="font-weight:normal">The Cooperative Institute for Mesoscale Meteorological
Studies</b> (OU/NOAA)<br>Los Alamos National Laboratory, Los Alamos, NM (LANL)<br><br>"Yesterday is History, Tomorrow is a Mystery and Today is a Gift; That is why it is called the Present" <br><br>"There are only 10 types of people in the world:<br>
Those who understand binary, and those who don't" <br><br>"My opinions are my own and not representative of OU, NSSL,<br>AOML, HRD, LANL or any affiliates."<br> ^.^<br> (o o)<br> /( V )\<br>
---m---m----<br><br><br><br><div class="gmail_quote"><div><div class="h5">On Mon, Mar 26, 2012 at 7:22 AM, Don Morton <span dir="ltr"><<a href="mailto:Don.Morton@alaska.edu" target="_blank">Don.Morton@alaska.edu</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="h5">
Howdy,<div><br></div><div>I suspect you have over-decomposed your Nest 2.</div><div><br></div><div>Your Nest 2 has 151x196 = 29,596 horizontal grid points. With 1152 tasks, each task only has about 26 grid points, or a 5x5 grid. At this level of refinement, I believe you're getting into issues of not having enough grid points for halos, etc.</div>
<div><br></div><div>Actually, even with 224 cores, you only have about 132 grid points, or an 11x11 grid in each task. Some have suggested in the past that maybe once you get below about 15x15 grid points per task, your scalability starts to suffer.</div>
<div><br></div><div>So, to re-answer your previous question, WRF will work with tens to hundreds of thousands of tasks, but you need to do this with sizable problems. You can only decompose a given problem size so much until </div>
<div><br></div><div>a) It just doesn't scale well anymore, and</div><div>b) You over-refine it so much that it won't even run. I suspect this is your problem with the 1152 tasks.</div><div><br></div><div>Best Regards,</div>
<div><br></div><div>Don Morton</div><span><font color="#888888"><div><br></div></font></span><div><span><font color="#888888"><br>-- <br><div>Voice: <a href="tel:%2B1%20907%20450%208679" value="+19074508679" target="_blank">+1 907 450 8679</a></div>
Arctic Region Supercomputing Center<br><a href="http://weather.arsc.edu/" target="_blank">http://weather.arsc.edu/</a><div>
<a href="http://www.arsc.edu/%7Emorton/" target="_blank">http://people.arsc.edu/~morton/</a></div></font></span><div><div><div><br></div><br><div class="gmail_quote">On Mon, Mar 26, 2012 at 8:42 AM, brick <span dir="ltr"><<a href="mailto:brickflying@gmail.com" target="_blank">brickflying@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi<div><br></div><div>Thanks for help.</div><div>Today I test wrf3.3 with 224 cores. It goes well. But when I increase cores to 1120, wrf.exe didn't integrate after 6 hours and it also didn't stop or return any error massage.</div>
<div>The rsl.out.0000 show that wrf.exe stop at deal with domain2. Last 20 lines of
rsl.out.0000 is shown here.</div><div><br><div>769 Timing for main: time 2012-03-22_05:57:30 on domain 1: 0.10120 elapsed seconds.</div><div>770 Timing for main: time 2012-03-22_05:58:00 on domain 1: 0.10280 elapsed seconds.</div>
<div>771 Timing for main: time 2012-03-22_05:58:30 on domain 1: 0.10070 elapsed seconds.</div><div>772 Timing for main: time 2012-03-22_05:59:00 on domain 1: 0.10150 elapsed seconds.</div><div>773 Timing for main: time 2012-03-22_05:59:30 on domain 1: 0.10080 elapsed seconds.</div>
<div>774 Timing for main: time 2012-03-22_06:00:00 on domain 1: 0.09900 elapsed seconds.</div><div>775 *************************************</div><div>776 Nesting domain</div><div>777 ids,ide,jds,jde 1 151 1 196</div>
<div>778 ims,ime,jms,jme -4 15 -4 20</div><div>779 ips,ipe,jps,jpe 1 5 1 6</div><div>780 INTERMEDIATE domain</div><div>781 ids,ide,jds,jde 243 278 150 194</div>
<div>782 ims,ime,jms,jme 238 255 145 162</div><div>783 ips,ipe,jps,jpe 241 245 148 152</div><div>784 *************************************</div><div>
785 d01 2012-03-22_06:00:00 alloc_space_field: domain 2,</div><div>786 18001632 bytes allocated</div><div>787 d01 2012-03-22_06:00:00 alloc_space_field: domain 2,</div><div>788 1941408 bytes allocated</div>
<div>789 d01 2012-03-22_06:00:00 *** Initializing nest domain # 2 from an input file. **</div><div>790 *</div><div>791 d01 2012-03-22_06:00:00 med_initialdata_input: calling input_input</div><div><br></div><div>The namelist is :</div>
<div><div> 1 &time_control</div><div> 2 run_days = 0,</div><div> 3 run_hours = 72,</div><div> 4 run_minutes = 0,</div><div> 5 run_seconds = 0,</div><div> 6 start_year = 2012, 2012,</div><div> 7 start_month = 03, 03,</div>
<div> 8 start_day = 22, 22,</div><div> 9 start_hour = 00, 06,</div><div> 10 start_minute = 00, 00,</div><div> 11 start_second = 00, 00,</div><div> 12 end_year = 2012, 2012,</div><div> 13 end_month = 03, 03,</div>
<div> 14 end_day = 25, 23,</div><div> 15 end_hour = 00, 06,</div><div> 16 end_minute = 00, 00,</div><div> 17 end_second = 00, 00,</div><div> 18 interval_seconds = 21600,</div><div> 19 input_from_file = .true.,.true.,</div>
<div> 20 history_interval = 60, 60,</div><div> 21 frames_per_outfile = 13,13,</div><div> 22 restart = .false.,</div><div> 23 restart_interval = 36000,</div><div> 24 io_form_history = 2,</div><div> 25 io_form_restart = 2,</div>
<div> 26 io_form_input = 2,</div><div> 27 io_form_boundary = 2,</div><div> 28 debug_level = 0,</div><div> 29 /</div><div> 30</div><div> 31 &domains</div><div> 32 time_step = 30,</div><div> 33 time_step_fract_num = 0,</div>
<div> 34 time_step_fract_den = 1,</div><div> 35 max_dom = 2,</div><div> 36 s_we = 1, 1, 1,</div><div> 37 e_we = 441, 151,</div><div> 38 s_sn = 1, 1, 1,</div><div> 39 e_sn = 369, 196,</div><div> 40 s_vert = 1, 1, 1,</div>
<div> 41 e_vert = 51,51,</div><div> 42 p_top_requested = 5000,</div><div> 43 num_metgrid_levels = 27,</div><div> 44 num_metgrid_soil_levels = 4,</div><div> 45 dx = 5000, 1000,</div><div> 46 dy = 5000, 1000,</div>
<div> 47 grid_id = 1, 2, 3,</div><div> 48 parent_id = 0, 1, 2,</div><div> 49 i_parent_start = 0, 245,</div><div> 50 j_parent_start = 0, 152,</div><div> 51 parent_grid_ratio = 1, 5,</div></div><div>
<div> 52 parent_time_step_ratio = 1, 5,</div><div> 53 feedback = 0,</div><div> 54 smooth_option = 0,</div><div> 55 /</div><div> 56</div><div> 57 &physics</div><div> 58 mp_physics = 6,6,</div><div> 59 ra_lw_physics = 1, 1, 1,</div>
<div> 60 ra_sw_physics = 1, 1, 1,</div><div> 61 radt = 5,1,</div><div> 62 sf_sfclay_physics = 1,1,</div><div> 63 sf_surface_physics = 2, 2, 2,</div><div> 64 bl_pbl_physics = 1,1,</div><div> 65 bldt = 0, 0, 0,</div>
<div> 66 cu_physics = 0,0,</div><div> 67 cudt = 5, 5, 5,</div><div> 68 isfflx = 1,</div><div> 69 ifsnow = 0,</div><div> 70 icloud = 1,</div><div> 71 surface_input_source = 1,</div><div> 72 num_soil_layers = 4,</div>
<div> 73 sf_urban_physics = 0, 0, 0,</div><div> 74 /</div><div> 75</div><div> 76 &fdda</div><div> 77 /</div><div> 78</div><div> 79 &dynamics</div><div> 80 w_damping = 0,</div><div> 81 diff_opt = 1,</div>
<div> 82 km_opt = 4,</div><div> 83 diff_6th_opt = 0, 0, 0,</div><div> 84 diff_6th_factor = 0.12, 0.12, 0.12,</div><div> 85 base_temp = 290.,</div><div> 86 damp_opt = 1,</div><div> 87 zdamp = 5000,</div>
<div>
88 dampcoef = 0.01,</div><div> 89 khdif = 0, 0, 0,</div><div> 90 kvdif = 0, 0, 0,</div><div> 91 non_hydrostatic = .true., .true., .true.,</div><div> 92 moist_adv_opt = 1, 1, 1,</div><div> 93 scalar_adv_opt = 1, 1, 1,</div>
<div> 94 /</div><div> 95</div><div> 96 &bdy_control</div><div> 97 spec_bdy_width = 5,</div><div> 98 spec_zone = 1,</div><div> 99 relax_zone = 4,</div><div>100 specified = .true., .false., .false.,</div>
<div>
101 nested = .false., .true., .true.,</div><div>102 /</div></div><div><div>103</div><div>104 &grib2</div><div>105 /</div><div>106</div><div>107 &namelist_quilt</div><div>108 nio_tasks_per_group = 0,</div>
<div>109 nio_groups = 1,</div><div>110 imelist_quilt</div><div>111 108 nio_tasks_per_group = 0,</div><div>112 109 nio_groups = 1,</div><div>113 110 /</div><div>114</div></div><div><br></div><div>Thanks a lot.</div>
<div><br></div><div>brick</div><div><br></div><div><br></div><div><br></div><div><br></div><div class="gmail_quote">On Sat, Mar 24, 2012 at 12:39 AM, Welsh, Patrick T <span dir="ltr"><<a href="mailto:pat.welsh@unf.edu" target="_blank">pat.welsh@unf.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<font face="Arial"><span style="font-size:11pt">It runs fine with hundreds, ok with thousands.<br>
<br>
Pat<div><div><div><div><br>
<br>
<br>
On 3/23/12 4:12 AM, "brick" <<a href="http://brickflying@gmail.com" target="_blank">brickflying@gmail.com</a>> wrote:<br>
<br>
</div></div></div></div></span></font><div><div><div><div><blockquote><font face="Arial"><span style="font-size:11pt">Hi All<br>
<br>
Is there a limit of core number that WRF could use? I plan test WRF with 2048 cores or more next week. Could WRF run with such huge number?<br>
Thanks a lot.<br>
<br>
brick<br>
<br>
</span></font></blockquote></div></div></div></div><font face="Arial"><span style="font-size:11pt"><br>
-- <br>
<br>
</span></font>
</div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>
</div></div></div>
<br></div></div><div class="im">_______________________________________________<br>
Wrf-users mailing list<br>
<a href="mailto:Wrf-users@ucar.edu" target="_blank">Wrf-users@ucar.edu</a><br>
<a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" target="_blank">http://mailman.ucar.edu/mailman/listinfo/wrf-users</a><br>
<br></div></blockquote></div><br><br clear="all"><br><br>
</blockquote></div><br></div>