<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hey folks – for my work I have WRF/ARW 3.3.1 running in an operational mode for forecasting purposes. We use GFS as the primary forcing data, and we begin our runs once we have 24 hours of GFS available (in order to get things underway quickly.) Once that portion is complete, and assuming we have the remainder of our desired GFS data available, we restart the run and continue our forecast (after running the usual WPS pre-processing, etc).<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Generally, this works quite well. Maybe once a week (ie, once every 30 runs or so, we run 4 runs a day, 7 days a week), the job hangs up shortly after beginning the restart. The job typically outputs the wrfout files for the first time step, and sends significant output to the rsl.error/rsl.out files. The processes still show as running, just no further output of any kind appears, and eventually the queue/batch system kills it once it goes beyond the allocated time. There are no error messages in any logs or rsl files or anywhere else I’ve seen. On rerunning the job, it nearly always runs to completion without problem.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Anyone seen this sort of thing? Until recently, I didn’t see this as a major problem, because by far the most important data was within the first 24 hours. Lately our business needs are making the longer-lead forecasts more valuable than they were. I suppose I can put in additional job-monitoring machinery, and attempt to “restart the restart” if it hangs up, but obviously I’d like to minimize the incidence in the first place.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’ve got WRF 3.5.1 running experimentally, but not enough yet to see if that helps. We try not to change our WRF version too frequently (or other components), since this is an operational system, and even slight changes can change behavior, skewing statistics, etc. But if it helped, I’d consider it.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Since it always seems to happen at the same point (after restart, after first wrfouts, before additional time stepping), I doubt it’s a hardware or systems issue. Maybe some infrequently triggered race condition or similar?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Other details: <o:p></o:p></p><p class=MsoNormal style='text-indent:.5in'>PGI 10.6<o:p></o:p></p><p class=MsoNormal> netcdf-4.1.3<o:p></o:p></p><p class=MsoNormal> mvapich2-1.7<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thoughts? Thanks,<o:p></o:p></p><p class=MsoNormal>Mike<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>-- <o:p></o:p></p><p class=MsoNormal>Mike Zulauf<o:p></o:p></p><p class=MsoNormal>Meteorologist, Lead Senior<o:p></o:p></p><p class=MsoNormal>Operational Meteorology <o:p></o:p></p><p class=MsoNormal>Iberdrola Renewables<o:p></o:p></p><p class=MsoNormal>1125 NW Couch, Suite 700<o:p></o:p></p><p class=MsoNormal>Portland, OR 97209<o:p></o:p></p><p class=MsoNormal>Office: 503-478-6304 Cell: 503-913-0403<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body><!--[object_id=#iberdrolaren.com#]--><FONT face=Tahoma size=2><FONT color=#0000ff><FONT color=#387000>
<P align=left><FONT face=Arial size=1>This message is intended for the exclusive attention of the recipient(s) indicated. Any information contained herein is strictly confidential and privileged. If you are not the intended recipient, please notify us by return e-mail and delete this message from your computer system. Any unauthorized use, reproduction, alteration, filing or sending of this message and/or any attached files may lead to legal action being taken against the party(ies) responsible for said unauthorized use. Any opinion expressed herein is solely that of the author(s) and does not necessarily represent the opinion of the Company. The sender does not guarantee the integrity, speed or safety of this message, and does not accept responsibility for any possible damage arising from the interception, incorporation of viruses, or any other damage as a result of manipulation.</P></FONT></FONT><FONT color=#387000></FONT><FONT face=Arial size=1>
<P align=left></P></FONT></FONT></FONT></html>