<div dir="ltr">I can concur that in the distant past, I had similar issues which were tracked down to the MPI stack. Seems like Open MPI was more stable at the time. These things typically happened during a scatter/gather type of operation, like the reading of a restart file, or an lbc file, or even just an output, where Task 0 is doing some heavy communication with all the other tasks.</div>
<div class="gmail_extra"><br clear="all"><div><div>Voice: +1 907 450 8679</div>Arctic Region Supercomputing Center<br><a href="http://weather.arsc.edu/" target="_blank">http://weather.arsc.edu/</a><div><a href="http://www.arsc.edu/%7Emorton/" target="_blank">http://people.arsc.edu/~morton/</a></div>
</div>
<br><br><div class="gmail_quote">On Mon, Feb 3, 2014 at 10:26 AM, Dominikus Heinzeller <span dir="ltr"><<a href="mailto:climbfuji@ymail.com" target="_blank">climbfuji@ymail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Michael and Dmitry,<br>
<br>
I encountered a similar problem with WRF/ARW 3.5 for the following combination, very similar to yours:<br>
<br>
pgi64-12.10<br>
netcdf-3.6.3 / netcdf-4.2.1.1<br>
mvapich2-1.9<br>
<br>
It went away when using mpich-3.0.2 instead of mvapich2. For other compilers (gnu, intel), this version of mvapich2 worked fine with any netcdf version.<br>
<br>
Dom<br>
<div class="HOEnZb"><div class="h5"><br>
On 31/01/2014, at 11:25 pm, Dmitry N. Mikushin <<a href="mailto:maemarcus@gmail.com">maemarcus@gmail.com</a>> wrote:<br>
<br>
> Hi Michael,<br>
><br>
> This looks like a nasty software bug, that needs a state comparison.<br>
> In order to get one, I'd try to modify the model code to signal<br>
> entering abnormal state as early as possible. In this code I'd also<br>
> put an infinite loop, such that engineer can get back on cluster and<br>
> attach the debugger to the processes of problematic run. Then -<br>
> keeping the problematic run on hold, I'd start another equal run in<br>
> another instance of debugger, let it work to the point where the first<br>
> run entered the fail-path. As result, you will have "bad" and "good"<br>
> runs frozen at the point of problem, and will be able to compare their<br>
> states interactively through debuggers.<br>
><br>
> - D.<br>
><br>
><br>
> 2014-01-31 Zulauf, Michael <<a href="mailto:Michael.Zulauf@iberdrolaren.com">Michael.Zulauf@iberdrolaren.com</a>>:<br>
>> Hey folks - for my work I have WRF/ARW 3.3.1 running in an operational mode<br>
>> for forecasting purposes. We use GFS as the primary forcing data, and we<br>
>> begin our runs once we have 24 hours of GFS available (in order to get<br>
>> things underway quickly.) Once that portion is complete, and assuming we<br>
>> have the remainder of our desired GFS data available, we restart the run and<br>
>> continue our forecast (after running the usual WPS pre-processing, etc).<br>
>><br>
>><br>
>><br>
>> Generally, this works quite well. Maybe once a week (ie, once every 30 runs<br>
>> or so, we run 4 runs a day, 7 days a week), the job hangs up shortly after<br>
>> beginning the restart. The job typically outputs the wrfout files for the<br>
>> first time step, and sends significant output to the rsl.error/rsl.out<br>
>> files. The processes still show as running, just no further output of any<br>
>> kind appears, and eventually the queue/batch system kills it once it goes<br>
>> beyond the allocated time. There are no error messages in any logs or rsl<br>
>> files or anywhere else I've seen. On rerunning the job, it nearly always<br>
>> runs to completion without problem.<br>
>><br>
>><br>
>><br>
>> Anyone seen this sort of thing? Until recently, I didn't see this as a<br>
>> major problem, because by far the most important data was within the first<br>
>> 24 hours. Lately our business needs are making the longer-lead forecasts<br>
>> more valuable than they were. I suppose I can put in additional<br>
>> job-monitoring machinery, and attempt to "restart the restart" if it hangs<br>
>> up, but obviously I'd like to minimize the incidence in the first place.<br>
>><br>
>><br>
>><br>
>> I've got WRF 3.5.1 running experimentally, but not enough yet to see if that<br>
>> helps. We try not to change our WRF version too frequently (or other<br>
>> components), since this is an operational system, and even slight changes<br>
>> can change behavior, skewing statistics, etc. But if it helped, I'd<br>
>> consider it.<br>
>><br>
>><br>
>><br>
>> Since it always seems to happen at the same point (after restart, after<br>
>> first wrfouts, before additional time stepping), I doubt it's a hardware or<br>
>> systems issue. Maybe some infrequently triggered race condition or similar?<br>
>><br>
>><br>
>><br>
>> Other details:<br>
>><br>
>> PGI 10.6<br>
>><br>
>> netcdf-4.1.3<br>
>><br>
>> mvapich2-1.7<br>
>><br>
>><br>
>><br>
>> Thoughts? Thanks,<br>
>><br>
>> Mike<br>
>><br>
>><br>
>><br>
>> --<br>
>><br>
>> Mike Zulauf<br>
>><br>
>> Meteorologist, Lead Senior<br>
>><br>
>> Operational Meteorology<br>
>><br>
>> Iberdrola Renewables<br>
>><br>
>> 1125 NW Couch, Suite 700<br>
>><br>
>> Portland, OR 97209<br>
>><br>
>> Office: 503-478-6304 Cell: 503-913-0403<br>
>><br>
>><br>
>><br>
>> This message is intended for the exclusive attention of the recipient(s)<br>
>> indicated. Any information contained herein is strictly confidential and<br>
>> privileged. If you are not the intended recipient, please notify us by<br>
>> return e-mail and delete this message from your computer system. Any<br>
>> unauthorized use, reproduction, alteration, filing or sending of this<br>
>> message and/or any attached files may lead to legal action being taken<br>
>> against the party(ies) responsible for said unauthorized use. Any opinion<br>
>> expressed herein is solely that of the author(s) and does not necessarily<br>
>> represent the opinion of the Company. The sender does not guarantee the<br>
>> integrity, speed or safety of this message, and does not accept<br>
>> responsibility for any possible damage arising from the interception,<br>
>> incorporation of viruses, or any other damage as a result of manipulation.<br>
>><br>
>><br>
>> _______________________________________________<br>
>> Wrf-users mailing list<br>
>> <a href="mailto:Wrf-users@ucar.edu">Wrf-users@ucar.edu</a><br>
>> <a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" target="_blank">http://mailman.ucar.edu/mailman/listinfo/wrf-users</a><br>
>><br>
> _______________________________________________<br>
> Wrf-users mailing list<br>
> <a href="mailto:Wrf-users@ucar.edu">Wrf-users@ucar.edu</a><br>
> <a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" target="_blank">http://mailman.ucar.edu/mailman/listinfo/wrf-users</a><br>
<br>
_______________________________________________<br>
Wrf-users mailing list<br>
<a href="mailto:Wrf-users@ucar.edu">Wrf-users@ucar.edu</a><br>
<a href="http://mailman.ucar.edu/mailman/listinfo/wrf-users" target="_blank">http://mailman.ucar.edu/mailman/listinfo/wrf-users</a><br>
</div></div></blockquote></div><br></div>