<div dir="ltr">Hi Adam,<div><br></div><div>There are a couple of things I can see right off the bat.</div><div><br></div><div>1.) I always average large files by reading in reading in the dimension I want to average over</div><div>1 at a time and then sum them, create counts and then divide. It is a bit tricky to get right, but it is faster than dim_avg.</div><div>Use 0 to hold places where the data is not valid. So you will need a value array and a count array. Before dividing,</div><div>set the count positions that are 0 to a fillvalue. This will save memory and time. This method would affect parts of the code that are similar to:</div><div><br></div><div><div>in = addfiles (files0,"r")</div><div> ListSetType(in,"cat")</div><div><br></div><div> lat0 = in[0]->lat</div><div> lon0 = in[0]->lon</div><div> hyam = in[0]->hyam ; read from a file the mid-layer coef</div><div> hybm = in[0]->hybm ; read from a file</div><div> hyai = in[0]->hyai ; read from a file the interface-layer coef</div><div> hybi = in[0]->hybi ; read from a file</div><div> nlat = dimsizes(lat0)</div><div> nlon = dimsizes(lon0)</div><div> nlevs = dimsizes(hyam)</div><div><br></div><div> pc = in[:]->PRECC</div><div> pl = in[:]->PRECL</div><div><br></div><div> delete(in)</div><div><br></div><div> LHl0 = pl*L*rhofw</div><div> LHc0 = pc*L*rhofw</div><div> delete(pc)</div><div> delete(pl)</div><div> zlhl0 = dim_avg(dim_avg_n(LHl0,0))</div><div> zlhc0 = dim_avg(dim_avg_n(LHc0,0))</div></div><div><br></div><div><br></div><div>2.) You only need to addfile(file,"r") and then loop over the history, reading in it 1 at a time so:</div><div><br></div><div>in = addfile(files0(np),"r")</div><div>ps0 = in->PS ; surface pressure [Pa]</div><div><br></div><div>becomes</div><div><br></div><div><div>in = addfile(ifile,"r")</div><div><br></div><div>do np = 0, nh.</div><div> ps0 = in->PS(np,:,:) ; surface pressure [Pa]</div></div><div>end do</div><div><br></div><div>Hope this helps,</div><div>/Marston</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><font color="#0000ff">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</font></div><div><font color="#0000ff">Marston S. Johnston, PhD</font></div><div><font color="#0000ff">Department of Earth Sciences</font></div><div><font color="#0000ff">University of Gothenburg</font><span style="color:rgb(0,0,255)">, Sweden</span></div><div><font color="#0000ff">Email: <a href="mailto:marston.johnston@gu.se" target="_blank">marston.johnston@gu.se</a> </font></div><div><font color="#0000ff">Phone: +46-31-7864901 <br></font></div><div><font color="#0000ff">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</font></div><div><font color="#0000ff">Only the fruitful thing is true!</font></div></div></div></div></div></div>
<br><div class="gmail_quote">On Sat, Jul 23, 2016 at 1:11 AM, Adam Herrington <span dir="ltr"><<a href="mailto:adam.herrington@stonybrook.edu" target="_blank">adam.herrington@stonybrook.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>Im getting a core-dump from an ncl script that I'm running on yellowstone. This is probably due to memory issues. The core-dump occurs after i load the very costly 28km global simulations. My usual remedy is to loop through each history file individually (there are ~60 history files for each model run) instead of using "addfiles", which reduces the size of the variables.</div><div><br></div><div>Unfortunately, I'm still getting the core-dump. I've attached my script, and would appreciate any suggestions on ways to cut down on memory. </div><div><br></div><div>I've never tried splitting the variables into smaller chunks, but I guess I will start trying to do this. Unless someone has a better suggestion?</div><div><br></div><div>Thanks!</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Adam</div></font></span></div>
<br>_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>