# [ncl-talk] Slow code

Michael Notaro mnotaro at wisc.edu
Tue Jan 12 12:48:36 MST 2016

```Thanks, Mary, for your further suggestions.

After I got Alan's first email which helped me reassess my code,

I modified my code to remove the year dimension from most variables

to make them more manageable.  Now the code

runs in 1 hour, rather than 1 day+.

Michael

begin

mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)

daily365=new((/365,20,141,217/),float) ; 365 day calendar data
daily365 at _FillValue=1e+35
daily365=1e+35

do iyr=0,19

data=new((/141,217,12,744/),float) ; hourly data
data at _FillValue=1e+35
data=1e+35

cnt=0
do im=0,11
prefix=(1980+iyr)+""+mns(im)
iy=b->iy
jx=b->jx
xlat=b->xlat
xlon=b->xlon
snow=b->snv ; liquid equiv of snow on ground
dims=dimsizes(snow)
nt=dims(0)
data(:,:,im,0:nt-1)=snow(iy|:,jx|:,time|:)
delete(snow)
delete(b)
delete(dims)
cnt=cnt+1
end do
data at _FillValue=1e+20

daily=new((/141,217,12,31/),float) ; daily data per month
daily at _FillValue=1e+35
daily=1e+35

cnt=0
do id=0,30
daily(:,:,:,id)=dim_avg(data(:,:,:,cnt:cnt+23)) ; convert hourly to daily data
cnt=cnt+24
end do

delete(data)

cnt=0
do im=0,11
do id=0,ndays(im)-1
daily365(cnt,iyr,:,:)=daily(:,:,im,id) ; convert daily data per month to 365 day calendar
cnt=cnt+1
end do
end do

delete(daily)

end do

daily212=new((/19,212,141,217/),float) ; 212 day calendar data for Sep-Mar
daily212 at _FillValue=1e+35
daily212=1e+35

do iyr=0,18
daily212(iyr,0:121,:,:)=daily365(243:364,iyr,:,:) ; retrieve Sep-Mar
daily212(iyr,122:211,:,:)=daily365(0:89,iyr+1,:,:)
end do

delete(daily365)

year=ispan(0,18,1)
year!0="year"
year&year=year

time=ispan(0,211,1)
time!0="time"
time&time=time

daily212!0="year"
daily212!1="time"
daily212!2="iy"
daily212!3="jx"
daily212&year=year
daily212&time=time
daily212&iy=iy
daily212&jx=jx

daily212 at long_name = "liquid snow water on ground"
daily212 at units = "kg m-2"
daily212 at coordinates="xlat xlon"
daily212 at grid_mapping = "rcm_map"

system("rm save_daily212_actual_snv_access_late20_faster.nc")
out->daily212=daily212
out->xlat=xlat
out->xlon=xlon
out->iy=iy
out->jx=jx

Michael Notaro
Associate Director
Nelson Institute Center for Climatic Research
Phone: (608) 261-1503
Email: mnotaro at wisc.edu

________________________________
From: Mary Haley <haley at ucar.edu>
Sent: Tuesday, January 12, 2016 1:34 PM
To: Alan Brammer
Cc: Michael Notaro; ncl-talk at ucar.edu
Subject: Re: [ncl-talk] Slow code

Hi folks,

These are all good suggestions.

Another thing that is expensive in NCL is reordering arrays with syntax like:

snow(iy|:,jx|:,time|:),

NCL makes a copy of the array when you do this, and it has to swap all the dimensions every time in the loop.

Isf reordering the array is absolutely necessary? I see that you are reordering and then calling "dim_avg_n". Since you are already using dim_avg_n, why not leave the array as is and just change the dimension you do the average on?

--Mary

On Tue, Jan 12, 2016 at 8:30 AM, Alan Brammer <abrammer at albany.edu<mailto:abrammer at albany.edu>> wrote:
Hi Michael,

I was going to suggest reshape that data array but it’s 20GB and is going to be unnecessarily slow whatever.  Do you actually need to store all the hourly data? the below edits suggest that you don’t.  The below uses less than a 1GB of memory rather than 20+GB.

This is obviously untested so may need editing.
(requires 6.1.1 or newer. )

mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)

daily=new((/141,217,20,12,31/),float) ; hourly data
daily at _FillValue=1e+35
daily=1e+35

cnt=0
do iyr=0,19
do im=0,11
prefix=(1980+iyr)+""+mns(im)
iy=b->iy
jx=b->jx
xlat=b->xlat   ; These aren’t doing anything?
xlon=b->xlon ; These aren’t doing anything?
snow =b->snv ; liquid equiv of snow on ground
dims=dimsizes(snow)
nt=dims(0)

snow4d := reshape( snow(iy|:,jx|:,time|:), (/dims(1), dims(2), ndays(im), 24/) ) ; I assume snow is originally (time|:,iy|:,ix|:)
daily(:,:,iyr,im,:ndays(im)-1)=dim_avg_n(snow4d, 3)

delete(snow)
delete(b)
delete(dims)
cnt=cnt+1
end do
end do

daily at _FillValue=1e+20

Good luck,

Alan Brammer.

On 12 Jan 2016, at 10:00, Michael Notaro <mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>> wrote:

Thanks for your email.

Actually, this is the main part slowing me down, not the top part of the code with the addfiles.

cnt=0
do id=0,30
daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to daily data     ***** THIS PART IS SLOW *****
cnt=cnt+24
end do

Any way to perform this task quicker?

Michael

Michael Notaro
Associate Director
Nelson Institute Center for Climatic Research
Phone: (608) 261-1503<tel:%28608%29%20261-1503>
Email: mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>

________________________________
From: Guido Cioni <guidocioni at gmail.com<mailto:guidocioni at gmail.com>>
Sent: Tuesday, January 12, 2016 8:57 AM
To: Michael Notaro
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Slow code

Everyone here will tell you that using loops in NCL it’s not efficient :)
But from my experience I think that the main thing slowing you down is that you are using addfile at every iteration.
Does creating a whole file and reading that in the beginning would change what you are trying to compute?

Guido Cioni
http://guidocioni.altervista.org<http://guidocioni.altervista.org/>

Guido Cioni<http://guidocioni.altervista.org/>
guidocioni.altervista.org<http://guidocioni.altervista.org/>
Le stazioni sono state riparate ed i dati vengono nuovamente aggiornati in tempo reale. Purtroppo a causa di numerosi malfunzionamenti i dati pluviometrici di Pisa e ...

On 12 Jan 2016, at 15:35, Michael Notaro <mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>> wrote:

Does anyone have a recommendation to speed up my code?
It's been running for a day now.  I put asterisks next to the real slow loop.
Basically, that part is converting a large array of hourly data into daily data.
Thanks, Michael

mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)

data=new((/141,217,20,12,744/),float) ; hourly data
data at _FillValue=1e+35
data=1e+35

cnt=0
do iyr=0,19
do im=0,11
prefix=(1980+iyr)+""+mns(im)
iy=b->iy
jx=b->jx
xlat=b->xlat
xlon=b->xlon
snow=b->snv ; liquid equiv of snow on ground
dims=dimsizes(snow)
nt=dims(0)
data(:,:,iyr,im,0:nt-1)=snow(iy|:,jx|:,time|:)
delete(snow)
delete(b)
delete(dims)
cnt=cnt+1
end do
end do
data at _FillValue=1e+20

daily=new((/141,217,20,12,31/),float) ; daily data per month
daily at _FillValue=1e+35
daily=1e+35

cnt=0
do id=0,30
daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to daily data     ***** THIS PART IS SLOW *****
cnt=cnt+24
end do

delete(data)

Michael Notaro
Associate Director
Nelson Institute Center for Climatic Research
Phone: (608) 261-1503<tel:%28608%29%20261-1503>
Email: mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/4ddf6a9d/attachment-0001.html
```