[ncl-talk] Slow code

Tue Jan 12 12:34:34 MST 2016

Hi folks,

These are all good suggestions.

Another thing that is expensive in NCL is reordering arrays with syntax
like:

snow(iy|:,jx|:,time|:),

NCL makes a copy of the array when you do this, and it has to swap all the
dimensions every time in the loop.

Isf reordering the array is absolutely necessary? I see that you are
reordering and then calling "dim_avg_n". Since you are already using
dim_avg_n, why not leave the array as is and just change the dimension you
do the average on?

--Mary

On Tue, Jan 12, 2016 at 8:30 AM, Alan Brammer <abrammer at albany.edu> wrote:

> Hi Michael,
>
>
> I was going to suggest reshape that data array but it’s 20GB and is going
> to be unnecessarily slow whatever.  Do you actually need to store all the
> hourly data? the below edits suggest that you don’t.  The below uses less
> than a 1GB of memory rather than 20+GB.
>
>  This is obviously untested so may need editing.
> (requires 6.1.1 or newer. )
>
>
> mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
> ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)
>
> daily=new((/141,217,20,12,31/),float) ; hourly data
> daily at _FillValue=1e+35
> daily=1e+35
>
> cnt=0
> do iyr=0,19
>   do im=0,11
>     prefix=(1980+iyr)+""+mns(im)
>
> b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"
> 0100.nc","r") ; read hourly SRF data
>     iy=b->iy
>     jx=b->jx
>     xlat=b->xlat   ; These aren’t doing anything?
>     xlon=b->xlon ; These aren’t doing anything?
>     snow =b->snv ; liquid equiv of snow on ground
>     dims=dimsizes(snow)
>     nt=dims(0)
>
>     snow4d := reshape( snow(iy|:,jx|:,time|:), (/dims(1), dims(2),
> ndays(im), 24/) ) ; I assume snow is originally (time|:,iy|:,ix|:)
>     daily(:,:,iyr,im,:ndays(im)-1)=dim_avg_n(snow4d, 3)
>
>     delete(snow)
>     delete(b)
>     delete(dims)
>     cnt=cnt+1
>   end do
> end do
>
> daily at _FillValue=1e+20
>
>
> Good luck,
>
>
> Alan Brammer.
>
>
>
>
> On 12 Jan 2016, at 10:00, Michael Notaro <mnotaro at wisc.edu> wrote:
>
> Thanks for your email.
>
> Actually, this is the main part slowing me down, not the top part of the
> code with the addfiles.
>
> cnt=0
> do id=0,30
>   daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to
> daily data     ***** THIS PART IS SLOW *****
>   cnt=cnt+24
> end do
>
>
> Any way to perform this task quicker?
>
> Michael
>
>
>
>
>
> Michael Notaro
> Associate Director
> Nelson Institute Center for Climatic Research
> University of Wisconsin-Madison
> Phone: (608) 261-1503
> Email: mnotaro at wisc.edu
>
>
> ------------------------------
> *From:* Guido Cioni <guidocioni at gmail.com>
> *Sent:* Tuesday, January 12, 2016 8:57 AM
> *To:* Michael Notaro
> *Cc:* ncl-talk at ucar.edu
> *Subject:* Re: [ncl-talk] Slow code
>
> Everyone here will tell you that using loops in NCL it’s not efficient :)
> But from my experience I think that the main thing slowing you down is
> that you are using addfile at every iteration.
> Does creating a whole file and reading that in the beginning would change
> what you are trying to compute?
>
> Guido Cioni
> http://guidocioni.altervista.org
> <http://guidocioni.altervista.org/>
> Guido Cioni <http://guidocioni.altervista.org/>
> guidocioni.altervista.org
> Le stazioni sono state riparate ed i dati vengono nuovamente aggiornati in
> tempo reale. Purtroppo a causa di numerosi malfunzionamenti i dati
> pluviometrici di Pisa e ...
>
>
> On 12 Jan 2016, at 15:35, Michael Notaro <mnotaro at wisc.edu> wrote:
>
> Does anyone have a recommendation to speed up my code?
> It's been running for a day now.  I put asterisks next to the real slow
> loop.
> Basically, that part is converting a large array of hourly data into daily
> data.
> Thanks, Michael
>
>
> mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
> ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)
>
> data=new((/141,217,20,12,744/),float) ; hourly data
> data at _FillValue=1e+35
> data=1e+35
>
> cnt=0
> do iyr=0,19
>   do im=0,11
>     prefix=(1980+iyr)+""+mns(im)
>
> b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"
> 0100.nc","r") ; read hourly SRF data
>     iy=b->iy
>     jx=b->jx
>     xlat=b->xlat
>     xlon=b->xlon
>     snow=b->snv ; liquid equiv of snow on ground
>     dims=dimsizes(snow)
>     nt=dims(0)
>     data(:,:,iyr,im,0:nt-1)=snow(iy|:,jx|:,time|:)
>     delete(snow)
>     delete(b)
>     delete(dims)
>     cnt=cnt+1
>   end do
> end do
> data at _FillValue=1e+20
>
> daily=new((/141,217,20,12,31/),float) ; daily data per month
> daily at _FillValue=1e+35
> daily=1e+35
>
> cnt=0
> do id=0,30
>   daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to
> daily data     ***** THIS PART IS SLOW *****
>   cnt=cnt+24
> end do
>
> delete(data)
>
>
>
>
> Michael Notaro
> Associate Director
> Nelson Institute Center for Climatic Research
> University of Wisconsin-Madison
> Phone: (608) 261-1503
> Email: mnotaro at wisc.edu
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/537ea12f/attachment.html 