[ncl-talk] Slow code

[Alan Brammer]

Hi Michael, 


I was going to suggest reshape that data array but it’s 20GB and is going to be unnecessarily slow whatever.  Do you actually need to store all the hourly data? the below edits suggest that you don’t.  The below uses less than a 1GB of memory rather than 20+GB.  

 This is obviously untested so may need editing. 
(requires 6.1.1 or newer. )


mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)

daily=new((/141,217,20,12,31/),float) ; hourly data
daily at _FillValue=1e+35
daily=1e+35

cnt=0
do iyr=0,19
  do im=0,11
    prefix=(1980+iyr)+""+mns(im)
    b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"0100.nc","r") ; read hourly SRF data
    iy=b->iy
    jx=b->jx
    xlat=b->xlat  					; These aren’t doing anything?
    xlon=b->xlon 					; These aren’t doing anything?
    snow =b->snv ; liquid equiv of snow on ground
    dims=dimsizes(snow)
    nt=dims(0)

    snow4d := reshape( snow(iy|:,jx|:,time|:), (/dims(1), dims(2), ndays(im), 24/) ) ; I assume snow is originally (time|:,iy|:,ix|:)
    daily(:,:,iyr,im,:ndays(im)-1)=dim_avg_n(snow4d, 3)

    delete(snow)
    delete(b)
    delete(dims)
    cnt=cnt+1
  end do
end do

daily at _FillValue=1e+20


Good luck, 


Alan Brammer.




> On 12 Jan 2016, at 10:00, Michael Notaro <mnotaro at wisc.edu> wrote:
> 
> Thanks for your email.
> 
> Actually, this is the main part slowing me down, not the top part of the code with the addfiles.
> 
>> cnt=0
>> do id=0,30
>>   daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to daily data     ***** THIS PART IS SLOW *****
>>   cnt=cnt+24
>> end do
> 
> Any way to perform this task quicker?
> 
> Michael
> 
> 
> 
> 
> 
> Michael Notaro
> Associate Director
> Nelson Institute Center for Climatic Research
> University of Wisconsin-Madison
> Phone: (608) 261-1503
> Email: mnotaro at wisc.edu <mailto:mnotaro at wisc.edu>
> 
> 
> From: Guido Cioni <guidocioni at gmail.com <mailto:guidocioni at gmail.com>>
> Sent: Tuesday, January 12, 2016 8:57 AM
> To: Michael Notaro
> Cc: ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
> Subject: Re: [ncl-talk] Slow code
>  
> Everyone here will tell you that using loops in NCL it’s not efficient :)
> But from my experience I think that the main thing slowing you down is that you are using addfile at every iteration. 
> Does creating a whole file and reading that in the beginning would change what you are trying to compute? 
> 
> Guido Cioni
> http://guidocioni.altervista.org <http://guidocioni.altervista.org/> 
>  <http://guidocioni.altervista.org/>	
> Guido Cioni <http://guidocioni.altervista.org/>
> guidocioni.altervista.org <http://guidocioni.altervista.org/>
> Le stazioni sono state riparate ed i dati vengono nuovamente aggiornati in tempo reale. Purtroppo a causa di numerosi malfunzionamenti i dati pluviometrici di Pisa e ...
> 
> 
>> On 12 Jan 2016, at 15:35, Michael Notaro <mnotaro at wisc.edu <mailto:mnotaro at wisc.edu>> wrote:
>> 
>> Does anyone have a recommendation to speed up my code?
>> It's been running for a day now.  I put asterisks next to the real slow loop.
>> Basically, that part is converting a large array of hourly data into daily data.
>> Thanks, Michael
>> 
>> 
>> mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
>> ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)
>> 
>> data=new((/141,217,20,12,744/),float) ; hourly data
>> data at _FillValue=1e+35
>> data=1e+35
>> 
>> cnt=0
>> do iyr=0,19
>>   do im=0,11
>>     prefix=(1980+iyr)+""+mns(im)
>>     b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"0100.nc","r") ; read hourly SRF data
>>     iy=b->iy
>>     jx=b->jx
>>     xlat=b->xlat
>>     xlon=b->xlon
>>     snow=b->snv ; liquid equiv of snow on ground
>>     dims=dimsizes(snow)
>>     nt=dims(0)
>>     data(:,:,iyr,im,0:nt-1)=snow(iy|:,jx|:,time|:)
>>     delete(snow)
>>     delete(b)
>>     delete(dims)
>>     cnt=cnt+1
>>   end do
>> end do
>> data at _FillValue=1e+20
>> 
>> daily=new((/141,217,20,12,31/),float) ; daily data per month
>> daily at _FillValue=1e+35
>> daily=1e+35
>> 
>> cnt=0
>> do id=0,30
>>   daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to daily data     ***** THIS PART IS SLOW *****
>>   cnt=cnt+24
>> end do
>> 
>> delete(data)
>> 
>> 
>> 
>> 
>> Michael Notaro
>> Associate Director
>> Nelson Institute Center for Climatic Research
>> University of Wisconsin-Madison
>> Phone: (608) 261-1503
>> Email: mnotaro at wisc.edu <mailto:mnotaro at wisc.edu>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/63ad6350/attachment.html