<div dir="ltr"><div class="gmail_default" style="font-size:small">Hi David,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Glad you found a solution. </div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">When you put (/ and /) around an array on the right side of the "=" and then assign this to a variable, this tells NCL not to copy any metadata. </div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">This can save some time, but I'm not sure how it saved that much time in your case, because it doesn't look like this particular line is inside a "do" loop. See the "hscopy.ncl" script that I attached, that shows the timings from copy metadata versus not copying it.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">The times I got were not that different:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">
<p class=""><font face="monospace, monospace">==========================================================<br>Elapsed time for full metadata copy = 0.164273 CPU seconds.<br>Elapsed time for no metadata copy = 0.163253 CPU seconds.<br>==========================================================</font></p></div><div class="gmail_default" style="font-size:small">Perhaps you can insert some similar timing calls in your code to find out where some other culprits might be. This won't necessarily address the memory issue, but it might help see where you are spending a lot of time, and then you can see if this is also memory intensive.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Is there more to this script than what you cut-and-pasted in the original email?</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">--Mary</div><div class="gmail_default" style="font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 9, 2015 at 4:17 AM, David Craig <span dir="ltr"><<a href="mailto:dcdavemail@gmail.com" target="_blank">dcdavemail@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
thanks for your input, deleting ix, iy didn't help as they are
overwritten every loop. However, changing the second line from,<span class=""><br>
<br>
rand_HS = HS_reorder(:,:,1:) <br>
<br></span>
to,<br>
<br>
rand_HS = (/HS_reorder(:,:,1:)/)<br>
<br>
did the trick. I guess the variable HS_reorder was growing in every
loop for some reason but I am not clear why forcing it to an array
fixed the issue.<span class="HOEnZb"><font color="#888888"><br>
<br>
David</font></span><div><div class="h5"><br>
<br>
<br>
<br>
<div>On 08/02/15 23:41, Mary Haley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="font-size:small">I think the
problem might be with the two calls to generate_sample_indices
inside the double do loop.</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">If I'm
reading your code correctly, ix and iy will be size 317 and
720 elements respectively.</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">The
generate_sample_indices function is actually an NCL function
(and not a built-in function) that creates an array of random
numbers and then sorts them. This means you are creating two
new arrays every time inside this do loop,and never freeing up
the space. Also, the sorting may cause the code to slow down.</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">As a start,
you might try deleting ix and iy after each iteration in the
loop. </div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><span style="font-size:12.8000001907349px"> do while(i.lt.n)</span><br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px"> j = 0</span><br style="font-size:12.8000001907349px">
<br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px"> do
while(j.lt.m) ; loop through time</span><br style="font-size:12.8000001907349px">
<br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px"> ;
randomise in space</span><br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px"> ix =
generate_sample_indices( dms(0), 0 )</span><br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px"> iy =
generate_sample_indices( dms(1), 0 )</span><br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px">
rand_HS(:,:,j) = rand_HS(ix, iy, j)</span><br style="font-size:12.8000001907349px">
<br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px"> j=j+1</span></font></div>
<div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><span style="font-size:12.8000001907349px"><br>
</span></font></div>
<div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> delete([/ix,iy/])<br style="font-size:12.8000001907349px">
<span style="font-size:12.8000001907349px"> end do</span></font><br>
</div>
<div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><span style="font-size:12.8000001907349px"><br>
</span></font></div>
<div class="gmail_default" style="font-size:small"><span style="font-size:12.8000001907349px"><font face="arial,
helvetica, sans-serif">You might also consider whether you
need to do this every time inside the do loop.</font></span></div>
<div class="gmail_default" style="font-size:small"><span style="font-size:12.8000001907349px"><font face="arial,
helvetica, sans-serif"><br>
</font></span></div>
<div class="gmail_default" style="font-size:small"><span style="font-size:12.8000001907349px"><font face="arial,
helvetica, sans-serif">Let ncl-talk know if you continue
to have problems with this code even after adding the
delete command.</font></span></div>
<div class="gmail_default" style="font-size:small"><span style="font-size:12.8000001907349px"><font face="arial,
helvetica, sans-serif"><br>
</font></span></div>
<div class="gmail_default" style="font-size:small"><span style="font-size:12.8000001907349px"><font face="arial,
helvetica, sans-serif">--Mary</font></span></div>
<div class="gmail_default" style="font-size:small"><span style="font-size:12.8000001907349px"><font face="arial,
helvetica, sans-serif"><br>
</font></span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Feb 8, 2015 at 6:25 AM, David
Craig <span dir="ltr"><<a href="mailto:dcdavemail@gmail.com" target="_blank">dcdavemail@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I have a loop in NCL that randomises an array and preforms
some<br>
operations on the result. However, it seems to be using too
much memory<br>
and I don't understand why as all variables are
preallocated. Does<br>
anyone know what the problem is, code is below.<br>
<br>
thanks,<br>
<br>
David<br>
<br>
<br>
n = 1000<br>
rand_HS = HS_reorder(:,:,1:) ; size 317x720x240<br>
dms = dimsizes(rand_HS)<br>
m = dms(2)<br>
; preallocate array for maximum coefficient values<br>
max_rand_ccr = new(n,float)<br>
i = 0<br>
<br>
do while(i.lt.n)<br>
j = 0<br>
<br>
do while(j.lt.m) ; loop through time<br>
<br>
; randomise in space<br>
ix = generate_sample_indices( dms(0), 0 )<br>
iy = generate_sample_indices( dms(1), 0 )<br>
rand_HS(:,:,j) = rand_HS(ix, iy, j)<br>
<br>
j=j+1<br>
end do<br>
<br>
;calculate max correlation coefficient<br>
max_rand_ccr(i) = max(abs(escorc(data2,rand_HS)))<br>
i=i+1<br>
end do<br>
_______________________________________________<br>
ncl-talk mailing list<br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>