[ncl-talk] loop memory issue

[Mary Haley]

Hi David,

Glad you found a solution.

When you put (/ and /) around an array on the right side of the "=" and
then assign this to a variable, this tells NCL not to copy any metadata.

This can save some time, but I'm not sure how it saved that much time in
your case, because it doesn't look like this particular line is inside a
"do" loop. See the "hscopy.ncl" script that I attached, that shows the
timings from copy metadata versus not copying it.

The times I got were not that different:

==========================================================
Elapsed time for full metadata copy = 0.164273 CPU seconds.
Elapsed time for no metadata copy   = 0.163253 CPU seconds.
==========================================================
Perhaps you can insert some similar timing calls in your code to find out
where some other culprits might be. This won't necessarily address the
memory issue, but it might help see where you are spending a lot of time,
and then you can see if this is also memory intensive.

Is there more to this script than what you cut-and-pasted in the original
email?

--Mary


On Mon, Feb 9, 2015 at 4:17 AM, David Craig <dcdavemail at gmail.com> wrote:

>  Hi,
>
> thanks for your input, deleting ix, iy didn't help as they are overwritten
> every loop. However, changing the second line from,
>
> rand_HS = HS_reorder(:,:,1:)
>
> to,
>
> rand_HS = (/HS_reorder(:,:,1:)/)
>
> did the trick. I guess the variable HS_reorder was growing in every loop
> for some reason but I am not clear why forcing it to an array fixed the
> issue.
>
> David
>
>
>
>
> On 08/02/15 23:41, Mary Haley wrote:
>
>  I think the problem might be with the two calls to
> generate_sample_indices inside the double do loop.
>
>  If I'm reading your code correctly, ix and iy will be size 317 and 720
> elements respectively.
>
>  The generate_sample_indices function is actually an NCL function (and
> not a built-in function) that creates an array of random numbers and then
> sorts them.  This means you are creating two new arrays every time inside
> this do loop,and never freeing up the space.  Also, the sorting may cause
> the code to slow down.
>
>  As a start, you might try deleting ix and iy after each iteration in the
> loop.
>
>     do while(i.lt.n)
>      j = 0
>
>      do while(j.lt.m) ; loop through time
>
>        ; randomise in space
>        ix = generate_sample_indices( dms(0), 0 )
>        iy = generate_sample_indices( dms(1), 0 )
>        rand_HS(:,:,j) = rand_HS(ix, iy, j)
>
>        j=j+1
>
>         delete([/ix,iy/])
>      end do
>
>  You might also consider whether you need to do this every time inside
> the do loop.
>
>  Let ncl-talk know if you continue to have problems with this code even
> after adding the delete command.
>
>  --Mary
>
>
> On Sun, Feb 8, 2015 at 6:25 AM, David Craig <dcdavemail at gmail.com> wrote:
>
>> Hi,
>>
>> I have a loop in NCL that randomises an array and preforms some
>> operations on the result. However, it seems to be using too much memory
>> and I don't understand why as all variables are preallocated. Does
>> anyone know what the problem is, code is below.
>>
>> thanks,
>>
>> David
>>
>>
>>    n = 1000
>>    rand_HS = HS_reorder(:,:,1:)     ; size 317x720x240
>>    dms = dimsizes(rand_HS)
>>    m = dms(2)
>>    ; preallocate array for maximum coefficient values
>>    max_rand_ccr = new(n,float)
>>    i = 0
>>
>>    do while(i.lt.n)
>>      j = 0
>>
>>      do while(j.lt.m) ; loop through time
>>
>>        ; randomise in space
>>        ix = generate_sample_indices( dms(0), 0 )
>>        iy  = generate_sample_indices( dms(1), 0 )
>>        rand_HS(:,:,j) = rand_HS(ix, iy, j)
>>
>>        j=j+1
>>      end do
>>
>>      ;calculate max correlation coefficient
>>      max_rand_ccr(i) = max(abs(escorc(data2,rand_HS)))
>>      i=i+1
>>    end do
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150209/0725d3d6/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hscopy.ncl
Type: application/octet-stream
Size: 1269 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150209/0725d3d6/attachment.obj