<div dir="ltr">It's pretty straightforward but you have to watch your FillValues and zero values.<div>This is for a case where 0 is not a valid data value. Otherwise you need a 3rd array to track those.</div><div><br></div><div>Let's say you have a do loop where you read each time step you want to average over:</div><div><br></div><div>Full data array X(time,nlat,nlon). You could read in all the data and then xavg = dim_avg_n_Wrap(X,(/0/))</div><div>but this is slow and memory intensive. Also you can track the sample size for each data point. </div><div><br></div><div>My suggestion is faster and requires far less memory. !!CAVE!! make sure you look at your data and know the valid ranges and invalid data. All of these must be accounted for before doing the cumulative averaging.</div><div> </div><div>a = addle(datafile,"r")</div><div><br></div><div>1.) Create an array to carry the cumulative sum and another to carry the counts: </div><div> do timestep=0,ntime</div><div> mh := a->X(timestep,:,:)</div><div> if(timestep.eq.0) then ; do only once</div><div> dims = dimsizes(mhm)</div><div> out = new(dims,"float")</div><div> outc = out</div><div> outc = 0.0</div><div> out = 0.0</div><div> endif</div><div> ; do the cumulative averaging</div><div> out = out + where(.not.ismissing(mhs),mhs,0.0) ; make sure your fillvalue is set properly. <br></div><div> outc = outc + where(.not.ismissing(mhs),1.0,0.0)</div><div> end do</div><div><br></div><div>2.) Set 0 values to FillValue to avoid divide by 0. If 0 is valid they will not add to the cumulative sum but some locations might </div><div>be 0 all the time and should not be replaced by a fillvalue as in the case below. In such cases, valid locations with 0 must be tracked with a unique array out_zero, e.g.</div><div><br></div><div>outc = where(outc.lt.1.0,default_fillvalue("float"),out) ; 0 is not a valid data value</div><div>ret = out/outc ; no divide by 0</div><div><br></div><div>3.) Replace Fillvalue with 0 if needed.</div><div><br></div><div> copy_VarMeta(mhs,ret)</div><div><br></div><div>Hope this helps,</div><div>/M</div><div> </div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><font color="#0000ff">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</font></div><div><font color="#0000ff">Marston S. Johnston, PhD</font></div><div><font color="#0000ff">Department of Earth Sciences</font></div><div><font color="#0000ff">University of Gothenburg</font><span style="color:rgb(0,0,255)">, Sweden</span></div><div><font color="#0000ff">Email: <a href="mailto:marston.johnston@gu.se" target="_blank">marston.johnston@gu.se</a> </font></div><div><font color="#0000ff">Phone: +46-31-7864901 <br></font></div><div><font color="#0000ff">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</font></div><div><font color="#0000ff">Only the fruitful thing is true!</font></div></div></div></div></div></div>
<br><div class="gmail_quote">On Sat, Jul 23, 2016 at 6:30 PM, adamrhster . <span dir="ltr"><<a href="mailto:adamrhster@gmail.com" target="_blank">adamrhster@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the suggestions. I get your 2nd point, but am having trouble following your 1st point about manually computing the average. Possible to give a quick example?<div><br></div><div>Adam</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 23, 2016 at 10:18 AM, Marston Johnston <span dir="ltr"><<a href="mailto:shejo284@gmail.com" target="_blank">shejo284@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Adam,<div><br></div><div>There are a couple of things I can see right off the bat.</div><div><br></div><div>1.) I always average large files by reading in reading in the dimension I want to average over</div><div>1 at a time and then sum them, create counts and then divide. It is a bit tricky to get right, but it is faster than dim_avg.</div><div>Use 0 to hold places where the data is not valid. So you will need a value array and a count array. Before dividing,</div><div>set the count positions that are 0 to a fillvalue. This will save memory and time. This method would affect parts of the code that are similar to:</div><div><br></div><div><div>in = addfiles (files0,"r")</div><div> ListSetType(in,"cat")</div><div><br></div><div> lat0 = in[0]->lat</div><div> lon0 = in[0]->lon</div><div> hyam = in[0]->hyam ; read from a file the mid-layer coef</div><div> hybm = in[0]->hybm ; read from a file</div><div> hyai = in[0]->hyai ; read from a file the interface-layer coef</div><div> hybi = in[0]->hybi ; read from a file</div><div> nlat = dimsizes(lat0)</div><div> nlon = dimsizes(lon0)</div><div> nlevs = dimsizes(hyam)</div><div><br></div><div> pc = in[:]->PRECC</div><div> pl = in[:]->PRECL</div><div><br></div><div> delete(in)</div><div><br></div><div> LHl0 = pl*L*rhofw</div><div> LHc0 = pc*L*rhofw</div><div> delete(pc)</div><div> delete(pl)</div><div> zlhl0 = dim_avg(dim_avg_n(LHl0,0))</div><div> zlhc0 = dim_avg(dim_avg_n(LHc0,0))</div></div><div><br></div><div><br></div><div>2.) You only need to addfile(file,"r") and then loop over the history, reading in it 1 at a time so:</div><div><br></div><div>in = addfile(files0(np),"r")</div><div>ps0 = in->PS ; surface pressure [Pa]</div><div><br></div><div>becomes</div><div><br></div><div><div>in = addfile(ifile,"r")</div><div><br></div><div>do np = 0, nh.</div><div> ps0 = in->PS(np,:,:) ; surface pressure [Pa]</div></div><div>end do</div><div><br></div><div>Hope this helps,</div><div>/Marston</div></div><div class="gmail_extra"><br clear="all"><div><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><font color="#0000ff">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</font></div><div><font color="#0000ff">Marston S. Johnston, PhD</font></div><div><font color="#0000ff">Department of Earth Sciences</font></div><div><font color="#0000ff">University of Gothenburg</font><span style="color:rgb(0,0,255)">, Sweden</span></div><div><font color="#0000ff">Email: <a href="mailto:marston.johnston@gu.se" target="_blank">marston.johnston@gu.se</a> </font></div><div><font color="#0000ff">Phone: <a href="tel:%2B46-31-7864901" value="+46317864901" target="_blank">+46-31-7864901</a> <br></font></div><div><font color="#0000ff">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</font></div><div><font color="#0000ff">Only the fruitful thing is true!</font></div></div></div></div></div></div>
<br><div class="gmail_quote"><div><div>On Sat, Jul 23, 2016 at 1:11 AM, Adam Herrington <span dir="ltr"><<a href="mailto:adam.herrington@stonybrook.edu" target="_blank">adam.herrington@stonybrook.edu</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi all,<div><br></div><div>Im getting a core-dump from an ncl script that I'm running on yellowstone. This is probably due to memory issues. The core-dump occurs after i load the very costly 28km global simulations. My usual remedy is to loop through each history file individually (there are ~60 history files for each model run) instead of using "addfiles", which reduces the size of the variables.</div><div><br></div><div>Unfortunately, I'm still getting the core-dump. I've attached my script, and would appreciate any suggestions on ways to cut down on memory. </div><div><br></div><div>I've never tried splitting the variables into smaller chunks, but I guess I will start trying to do this. Unless someone has a better suggestion?</div><div><br></div><div>Thanks!</div><span><font color="#888888"><div><br></div><div>Adam</div></font></span></div>
<br></div></div>_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>