Howdy,<div><br></div><div>I suspect you have over-decomposed your Nest 2.</div><div><br></div><div>Your Nest 2 has 151x196 = 29,596 horizontal grid points. With 1152 tasks, each task only has about 26 grid points, or a 5x5 grid. At this level of refinement, I believe you're getting into issues of not having enough grid points for halos, etc.</div>
<div><br></div><div>Actually, even with 224 cores, you only have about 132 grid points, or an 11x11 grid in each task. Some have suggested in the past that maybe once you get below about 15x15 grid points per task, your scalability starts to suffer.</div>
<div><br></div><div>So, to re-answer your previous question, WRF will work with tens to hundreds of thousands of tasks, but you need to do this with sizable problems. You can only decompose a given problem size so much until </div>
<div><br></div><div>a) It just doesn't scale well anymore, and</div><div>b) You over-refine it so much that it won't even run. I suspect this is your problem with the 1152 tasks.</div><div><br></div><div>Best Regards,</div>
<div><br></div><div>Don Morton</div><div><br></div><div><br class="Apple-interchange-newline">-- <br><div>Voice: +1 907 450 8679</div>Arctic Region Supercomputing Center<br><a href="http://weather.arsc.edu/" target="_blank">http://weather.arsc.edu/</a><div>
<a href="http://www.arsc.edu/~morton/" target="_blank">http://people.arsc.edu/~morton/</a></div><div><br></div><br><div class="gmail_quote">On Mon, Mar 26, 2012 at 8:42 AM, brick <span dir="ltr"><<a href="mailto:brickflying@gmail.com">brickflying@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi<div><br></div><div>Thanks for help.</div><div>Today I test wrf3.3 with 224 cores. It goes well. But when I increase cores to 1120, wrf.exe didn't integrate after 6 hours and it also didn't stop or return any error massage.</div>
<div>The rsl.out.0000 show that wrf.exe stop at deal with domain2. Last 20 lines of
rsl.out.0000 is shown here.</div><div><br><div>769 Timing for main: time 2012-03-22_05:57:30 on domain 1: 0.10120 elapsed seconds.</div><div>770 Timing for main: time 2012-03-22_05:58:00 on domain 1: 0.10280 elapsed seconds.</div>
<div>771 Timing for main: time 2012-03-22_05:58:30 on domain 1: 0.10070 elapsed seconds.</div><div>772 Timing for main: time 2012-03-22_05:59:00 on domain 1: 0.10150 elapsed seconds.</div><div>773 Timing for main: time 2012-03-22_05:59:30 on domain 1: 0.10080 elapsed seconds.</div>
<div>774 Timing for main: time 2012-03-22_06:00:00 on domain 1: 0.09900 elapsed seconds.</div><div>775 *************************************</div><div>776 Nesting domain</div><div>777 ids,ide,jds,jde 1 151 1 196</div>
<div>778 ims,ime,jms,jme -4 15 -4 20</div><div>779 ips,ipe,jps,jpe 1 5 1 6</div><div>780 INTERMEDIATE domain</div><div>781 ids,ide,jds,jde 243 278 150 194</div>
<div>782 ims,ime,jms,jme 238 255 145 162</div><div>783 ips,ipe,jps,jpe 241 245 148 152</div><div>784 *************************************</div><div>
785 d01 2012-03-22_06:00:00 alloc_space_field: domain 2,</div><div>786 18001632 bytes allocated</div><div>787 d01 2012-03-22_06:00:00 alloc_space_field: domain 2,</div><div>788 1941408 bytes allocated</div>
<div>789 d01 2012-03-22_06:00:00 *** Initializing nest domain # 2 from an input file. **</div><div>790 *</div><div>791 d01 2012-03-22_06:00:00 med_initialdata_input: calling input_input</div><div><br></div><div>The namelist is :</div>
<div><div> 1 &time_control</div><div> 2 run_days = 0,</div><div> 3 run_hours = 72,</div><div> 4 run_minutes = 0,</div><div> 5 run_seconds = 0,</div><div> 6 start_year = 2012, 2012,</div><div> 7 start_month = 03, 03,</div>
<div> 8 start_day = 22, 22,</div><div> 9 start_hour = 00, 06,</div><div> 10 start_minute = 00, 00,</div><div> 11 start_second = 00, 00,</div><div> 12 end_year = 2012, 2012,</div><div> 13 end_month = 03, 03,</div>
<div> 14 end_day = 25, 23,</div><div> 15 end_hour = 00, 06,</div><div> 16 end_minute = 00, 00,</div><div> 17 end_second = 00, 00,</div><div> 18 interval_seconds = 21600,</div><div> 19 input_from_file = .true.,.true.,</div>
<div> 20 history_interval = 60, 60,</div><div> 21 frames_per_outfile = 13,13,</div><div> 22 restart = .false.,</div><div> 23 restart_interval = 36000,</div><div> 24 io_form_history = 2,</div><div> 25 io_form_restart = 2,</div>
<div> 26 io_form_input = 2,</div><div> 27 io_form_boundary = 2,</div><div> 28 debug_level = 0,</div><div> 29 /</div><div> 30</div><div> 31 &domains</div><div> 32 time_step = 30,</div><div> 33 time_step_fract_num = 0,</div>
<div> 34 time_step_fract_den = 1,</div><div> 35 max_dom = 2,</div><div> 36 s_we = 1, 1, 1,</div><div> 37 e_we = 441, 151,</div><div> 38 s_sn = 1, 1, 1,</div><div> 39 e_sn = 369, 196,</div><div> 40 s_vert = 1, 1, 1,</div>
<div> 41 e_vert = 51,51,</div><div> 42 p_top_requested = 5000,</div><div> 43 num_metgrid_levels = 27,</div><div> 44 num_metgrid_soil_levels = 4,</div><div> 45 dx = 5000, 1000,</div><div> 46 dy = 5000, 1000,</div>
<div> 47 grid_id = 1, 2, 3,</div><div> 48 parent_id = 0, 1, 2,</div><div> 49 i_parent_start = 0, 245,</div><div> 50 j_parent_start = 0, 152,</div><div> 51 parent_grid_ratio = 1, 5,</div></div><div>
<div> 52 parent_time_step_ratio = 1, 5,</div><div> 53 feedback = 0,</div><div> 54 smooth_option = 0,</div><div> 55 /</div><div> 56</div><div> 57 &physics</div><div> 58 mp_physics = 6,6,</div><div> 59 ra_lw_physics = 1, 1, 1,</div>
<div> 60 ra_sw_physics = 1, 1, 1,</div><div> 61 radt = 5,1,</div><div> 62 sf_sfclay_physics = 1,1,</div><div> 63 sf_surface_physics = 2, 2, 2,</div><div> 64 bl_pbl_physics = 1,1,</div><div> 65 bldt = 0, 0, 0,</div>
<div> 66 cu_physics = 0,0,</div><div> 67 cudt = 5, 5, 5,</div><div> 68 isfflx = 1,</div><div> 69 ifsnow = 0,</div><div> 70 icloud = 1,</div><div> 71 surface_input_source = 1,</div><div> 72 num_soil_layers = 4,</div>
<div> 73 sf_urban_physics = 0, 0, 0,</div><div> 74 /</div><div> 75</div><div> 76 &fdda</div><div> 77 /</div><div> 78</div><div> 79 &dynamics</div><div> 80 w_damping = 0,</div><div> 81 diff_opt = 1,</div>
<div> 82 km_opt = 4,</div><div> 83 diff_6th_opt = 0, 0, 0,</div><div> 84 diff_6th_factor = 0.12, 0.12, 0.12,</div><div> 85 base_temp = 290.,</div><div> 86 damp_opt = 1,</div><div> 87 zdamp = 5000,</div>
<div>
88 dampcoef = 0.01,</div><div> 89 khdif = 0, 0, 0,</div><div> 90 kvdif = 0, 0, 0,</div><div> 91 non_hydrostatic = .true., .true., .true.,</div><div> 92 moist_adv_opt = 1, 1, 1,</div><div> 93 scalar_adv_opt = 1, 1, 1,</div>
<div> 94 /</div><div> 95</div><div> 96 &bdy_control</div><div> 97 spec_bdy_width = 5,</div><div> 98 spec_zone = 1,</div><div> 99 relax_zone = 4,</div><div>100 specified = .true., .false., .false.,</div>
<div>
101 nested = .false., .true., .true.,</div><div>102 /</div></div><div><div>103</div><div>104 &grib2</div><div>105 /</div><div>106</div><div>107 &namelist_quilt</div><div>108 nio_tasks_per_group = 0,</div>
<div>109 nio_groups = 1,</div><div>110 imelist_quilt</div><div>111 108 nio_tasks_per_group = 0,</div><div>112 109 nio_groups = 1,</div><div>113 110 /</div><div>114</div></div><div><br></div><div>Thanks a lot.</div>
<div><br></div><div>brick</div><div><br></div><div><br></div><div><br></div><div><br></div><div class="gmail_quote">On Sat, Mar 24, 2012 at 12:39 AM, Welsh, Patrick T <span dir="ltr"><<a href="mailto:pat.welsh@unf.edu" target="_blank">pat.welsh@unf.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<font face="Arial"><span style="font-size:11pt">It runs fine with hundreds, ok with thousands.<br>
<br>
Pat<div><div class="h5"><div><div><br>
<br>
<br>
On 3/23/12 4:12 AM, "brick" <<a href="http://brickflying@gmail.com" target="_blank">brickflying@gmail.com</a>> wrote:<br>
<br>
</div></div></div></div></span></font><div><div class="h5"><div><div><blockquote><font face="Arial"><span style="font-size:11pt">Hi All<br>
<br>
Is there a limit of core number that WRF could use? I plan test WRF with 2048 cores or more next week. Could WRF run with such huge number?<br>
Thanks a lot.<br>
<br>
brick<br>
<br>
</span></font></blockquote></div></div></div></div><font face="Arial"><span style="font-size:11pt"><br>
-- <br>
<br>
</span></font>
</div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>
</div>