Hello Michael and all,<br><br>Would just like to add that we are seeing the same problem with dmpar/PGI compiled WRF 3.2. We submit hundreds of jobs, and it seems that on about 3 or 4% of the jobs, they hang with no output, no error message, no crashes. The job ends up getting killed by the queueing system when it runs out of walltime. We are using PGI/Linux on Gentoo. The workaround has been to resubmit the hung jobs, which seems to fix the problem most (perhaps 96 to 97%?) of the time.<br>
<br>Previous versions of WRF 3.1 worked perfectly for us as well.<br><br>Let me know if any additional information is needed from us.<br><br>Best wishes,<br><br>Jeff<br><br><div class="gmail_quote">On Fri, Apr 30, 2010 at 2:01 PM, Zulauf, Michael <span dir="ltr"><<a href="mailto:Michael.Zulauf@iberdrolausa.com">Michael.Zulauf@iberdrolausa.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi again, all. . .<br>
<br>
I'm reviving my plea for help from a couple weeks ago. I'm still having<br>
issues with WRF 3.2 - and _only_ 3.2.<br>
<br>
I've tried different versions of the PGI compilers, different versions<br>
of support libraries, different optimization levels (all the way down to<br>
none), etc. My jobs sporadically (but usually eventually) hang up, most<br>
often after a new wrfout file is opened. No error messages, no crashes<br>
- the processes continue, but _all_ output stops. I eventually just<br>
have to kill the job. The wrfouts are small, and all output looks good<br>
up until the failed wrfout.<br>
<br>
The exact same hardware, OS, compilers, libraries, etc work for previous<br>
versions of WRF.<br>
<br>
Below is an example namelist.input (WPS seems to be running fine). Any<br>
thoughts?<br>
<br>
Thanks,<br>
Mike<br>
<br>
------------------------------------------------------------------------<br>
----------------------------<br>
&time_control<br>
run_days = 0,<br>
run_hours = 24,<br>
run_minutes = 0,<br>
run_seconds = 0,<br>
start_year = 2009,2009,2009,2009,<br>
start_month = 12,12,12,12,<br>
start_day = 14,14,14,14,<br>
start_hour = 00,03,06,09,<br>
start_minute = 00, 00, 00, 00, 00, 00,<br>
start_second = 00, 00, 00, 00, 00, 00,<br>
<br>
end_year = 2009,2009,2009,2009,<br>
end_month = 12,12,12,12,<br>
end_day = 15,15,15,14,<br>
end_hour = 00,00,00,12,<br>
end_minute = 00, 00, 00, 00, 00, 00,<br>
end_second = 00, 00, 00, 00, 00, 00,<br>
interval_seconds = 10800,<br>
input_from_file =<br>
.true.,.true.,.true.,.true.,.true.,<br>
fine_input_stream = 0, 2, 2, 2,<br>
io_form_auxinput2 = 2<br>
history_interval = 60,60,60,20,<br>
frames_per_outfile = 1, 1, 1, 1, 1, 1,<br>
restart = .false.,<br>
restart_interval = 1440,<br>
io_form_history = 2<br>
io_form_restart = 2<br>
io_form_input = 2<br>
io_form_boundary = 2<br>
debug_level = 0<br>
adjust_output_times = .true.<br>
/<br>
<br>
&domains<br>
time_step = 163,<br>
time_step_fract_num = 7,<br>
time_step_fract_den = 11,<br>
max_dom = 4,<br>
s_we = 1, 1, 1, 1, 1, 1,<br>
e_we = 142,244,280,382,<br>
s_sn = 1, 1, 1, 1, 1, 1,<br>
e_sn = 154,268,250,196,<br>
s_vert = 1, 1, 1, 1, 1, 1,<br>
e_vert = 31, 31, 31, 31, 31, 31,<br>
num_metgrid_levels = 27 ,<br>
eta_levels = 1.000, 0.993, 0.980, 0.966,<br>
0.950, 0.933, 0.913, 0.892, 0.869, 0.844, 0.816, 0.786, 0.753, 0.718,<br>
0.680, 0.639, 0.596, 0.550, 0.501, 0.451, 0.398, 0.345, 0.290, 0.236,<br>
0.188, 0.145, 0.108, 0.075, 0.046, 0.021, 0.000,<br>
<br>
p_top_requested = 5000,<br>
dx = 27000,9000,3000,1000,<br>
dy = 27000,9000,3000,1000,<br>
grid_id = 1, 2, 3, 4, 5, 6,<br>
parent_id = 1, 1, 2, 3, 4, 5,<br>
i_parent_start = 1,31,91,92,<br>
j_parent_start = 1,33,93,93,<br>
parent_grid_ratio = 1, 3, 3, 3, 3, 3,<br>
parent_time_step_ratio = 1, 3, 3, 3, 3, 3,<br>
feedback = 0,<br>
smooth_option = 2<br>
use_adaptive_time_step = .false.<br>
step_to_output_time = .true.<br>
target_cfl = 1.1,1.1,1.1,1.1,<br>
max_step_increase_pct = 5, 51, 51, 51, 51, 51<br>
starting_time_step = 162, 54, 18, 6<br>
max_time_step = 202.5, 67.5, 22.5, 7.5<br>
min_time_step = 27, 9, 3, 1<br>
adaptation_domain = 4<br>
/<br>
<br>
&physics<br>
mp_physics = 5, 5, 5, 5,<br>
ra_lw_physics = 1, 1, 1, 1,<br>
ra_sw_physics = 1, 1, 1, 1,<br>
radt = 30, 30, 30, 30, 30,<br>
30,<br>
sf_sfclay_physics = 1, 1, 1, 1,<br>
sf_surface_physics = 1, 1, 1, 1,<br>
bl_pbl_physics = 1, 1, 1, 1,<br>
bldt = 0, 0, 0, 0, 0,<br>
0,<br>
cu_physics = 1, 1, 0, 0, 0,<br>
0,<br>
cudt = 5, 5, 5, 0, 0,<br>
0,<br>
cam_abs_freq_s = 21600,<br>
levsiz = 59,<br>
paerlev = 29,<br>
cam_abs_dim1 = 4,<br>
cam_abs_dim2 = 31,<br>
isfflx = 1,<br>
ifsnow = 0,<br>
icloud = 1,<br>
surface_input_source = 1,<br>
num_soil_layers = 5,<br>
sf_urban_physics = 0, 0, 0, 0,<br>
mp_zero_out = 0,<br>
maxiens = 1,<br>
maxens = 3,<br>
maxens2 = 3,<br>
maxens3 = 16,<br>
ensdim = 144,<br>
slope_rad = 0,<br>
topo_shading = 0,<br>
/<br>
<br>
&fdda<br>
grid_fdda = 1, 0, 0,<br>
gfdda_inname = "wrffdda_d<domain>",<br>
gfdda_interval_m = 180, 0, 0,<br>
gfdda_end_h = 12, 0, 0,<br>
io_form_gfdda = 2,<br>
fgdt = 0, 0, 0,<br>
if_no_pbl_nudging_uv = 0, 0, 0,<br>
if_no_pbl_nudging_t = 1, 0, 0,<br>
if_no_pbl_nudging_q = 1, 0, 0,<br>
if_zfac_uv = 0, 0, 0,<br>
k_zfac_uv = 10, 10, 10,<br>
if_zfac_t = 1, 0, 0,<br>
k_zfac_t = 10, 10, 10,<br>
if_zfac_q = 1, 0, 0,<br>
k_zfac_q = 10, 10, 10,<br>
guv = 0.0001, 0.0001, 0.0001,<br>
gt = 0.0001, 0.0001, 0.0001,<br>
gq = 0.000001, 0.000001, 0.000001,<br>
if_ramping = 0,<br>
dtramp_min = 0.0,<br>
/<br>
<br>
&dynamics<br>
w_damping = 1,<br>
diff_opt = 1,<br>
km_opt = 4,<br>
diff_6th_opt = 0,<br>
diff_6th_factor = 0.12,<br>
base_temp = 290.<br>
damp_opt = 0,<br>
zdamp = 5000., 5000., 5000.,<br>
dampcoef = 0.01, 0.01, 0.01<br>
khdif = 0, 0, 0,<br>
kvdif = 0, 0, 0,<br>
non_hydrostatic = .true., .true., .true.,<br>
moist_adv_opt = 1, 1, 1, 1<br>
scalar_adv_opt = 1, 1, 1, 1<br>
use_baseparam_fr_nml = .true.<br>
/<br>
<br>
&bdy_control<br>
spec_bdy_width = 5,<br>
spec_zone = 1,<br>
relax_zone = 4,<br>
specified = .true.,<br>
.false.,.false.,.false.,.false., .false.,<br>
nested = .false., .true., .true.,.true.,<br>
.true., .true.,<br>
/<br>
<br>
&grib2<br>
/<br>
<br>
&namelist_quilt<br>
nio_tasks_per_group = 0,<br>
nio_groups = 1,<br>
/<br>
------------------------------------------------------------------------<br>
----------------------------<br>
<br>
-----Original Message-----<br>
Date: Fri, 16 Apr 2010 10:11:22 -0700<br>
From: "Zulauf, Michael" <<a href="mailto:Michael.Zulauf@iberdrolausa.com">Michael.Zulauf@iberdrolausa.com</a>><br>
Subject: Re: [Wrf-users] WRF 3.2 jobs hanging up sporadically on<br>
wrfout output<br>
To: "Don Morton" <<a href="mailto:Don.Morton@alaska.edu">Don.Morton@alaska.edu</a>><br>
Cc: <a href="mailto:wrf-users@ucar.edu">wrf-users@ucar.edu</a><br>
Message-ID:<br>
<br>
<<a href="mailto:B2A259FAA3CF26469FF9A7C7402C49970913EFE0@POREXUW03.ppmenergy.us">B2A259FAA3CF26469FF9A7C7402C49970913EFE0@POREXUW03.ppmenergy.us</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Thanks for the response, Don.<br>
<br>
The specific RDMA suggestion isn't relevant to our case (our hardware<br>
doesn't support it), but you may be right that this is an optimizations<br>
related issue. I'll probably try playing with optimizations next. I've<br>
got the same settings as has worked for previous versions - but perhaps<br>
something in the new code has made one of the settings problematic.<br>
<br>
Regarding the suggestions I've been getting relating to<br>
WRFIO_NCD_LARGE_FILE_SUPPORT - I don't think that's the problem. I'm<br>
splitting my output into single frame files to keep the file size small.<br>
I may try that also, just for the heck of it.<br>
<br>
Based on the sporadic nature of this (sometimes it happens, sometimes it<br>
doesn't, when it hangs seems fairly random), I suspect it's some type of<br>
timing issue like a race condition. If I can't get it working, I may<br>
just drop back to 3.1.1, at least until 3.2.1 comes out. ;-)<br>
<br>
Thanks all,<br>
<br>
Mike<br>
<br>
<br>
<br>
<br>
<br>
This message is intended for the exclusive attention of the address(es) indicated. Any information contained herein is strictly confidential and privileged, especially as regards person data,<br>
which must not be disclosed. If you are the intended recipient and have received it by mistake or learn about it in any other way, please notify us by return e-mail and delete this message from<br>
your computer system. Any unauthorized use, reproduction, alteration, filing or sending of this message and/or any attached files to third parties may lead to legal proceedings being taken. Any<br>
opinion expressed herein is solely that of the author(s) and does not necessarily represent the opinion of Iberdrola. The sender does not guarantee the integrity, speed or safety of this<br>
message, not accept responsibility for any possible damage arising from the interception, incorporation of virus or any other manipulation carried out by third parties.<br>
<br>
_______________________________________________<br>
Wrf-users mailing list<br>
<a href="mailto:Wrf-users@ucar.edu">Wrf-users@ucar.edu</a><br>
<a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" target="_blank">http://mailman.ucar.edu/mailman/listinfo/wrf-users</a><br>
</blockquote></div><br>