[ncl-talk] Slow code

Tue Jan 12 08:34:00 MST 2016

Thank you, from a fellow SUNY Albany alumni!  Using reshape is a creative idea.

Michael

Michael Notaro
Associate Director
Nelson Institute Center for Climatic Research
University of Wisconsin-Madison
Phone: (608) 261-1503
Email: mnotaro at wisc.edu

________________________________
From: Alan Brammer <abrammer at albany.edu>
Sent: Tuesday, January 12, 2016 9:30 AM
To: Michael Notaro
Cc: ncl-talk at ucar.edu
Subject: Re: [ncl-talk] Slow code

Hi Michael,

I was going to suggest reshape that data array but it’s 20GB and is going to be unnecessarily slow whatever.  Do you actually need to store all the hourly data? the below edits suggest that you don’t.  The below uses less than a 1GB of memory rather than 20+GB.

 This is obviously untested so may need editing.
(requires 6.1.1 or newer. )

mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)

daily=new((/141,217,20,12,31/),float) ; hourly data
daily at _FillValue=1e+35
daily=1e+35

cnt=0
do iyr=0,19
  do im=0,11
    prefix=(1980+iyr)+""+mns(im)
    b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"0100.nc","r") ; read hourly SRF data
    iy=b->iy
    jx=b->jx
    xlat=b->xlat   ; These aren’t doing anything?
    xlon=b->xlon ; These aren’t doing anything?
    snow =b->snv ; liquid equiv of snow on ground
    dims=dimsizes(snow)
    nt=dims(0)

    snow4d := reshape( snow(iy|:,jx|:,time|:), (/dims(1), dims(2), ndays(im), 24/) ) ; I assume snow is originally (time|:,iy|:,ix|:)
    daily(:,:,iyr,im,:ndays(im)-1)=dim_avg_n(snow4d, 3)

    delete(snow)
    delete(b)
    delete(dims)
    cnt=cnt+1
  end do
end do

daily at _FillValue=1e+20

Good luck,

Alan Brammer.

On 12 Jan 2016, at 10:00, Michael Notaro <mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>> wrote:

Thanks for your email.

Actually, this is the main part slowing me down, not the top part of the code with the addfiles.

cnt=0
do id=0,30
  daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to daily data     ***** THIS PART IS SLOW *****
  cnt=cnt+24
end do

Any way to perform this task quicker?

Michael

Michael Notaro
Associate Director
Nelson Institute Center for Climatic Research
University of Wisconsin-Madison
Phone: (608) 261-1503
Email: mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>

________________________________
From: Guido Cioni <guidocioni at gmail.com<mailto:guidocioni at gmail.com>>
Sent: Tuesday, January 12, 2016 8:57 AM
To: Michael Notaro
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Slow code

Everyone here will tell you that using loops in NCL it’s not efficient :)
But from my experience I think that the main thing slowing you down is that you are using addfile at every iteration.
Does creating a whole file and reading that in the beginning would change what you are trying to compute?

Guido Cioni
http://guidocioni.altervista.org<http://guidocioni.altervista.org/>
Guido Cioni<http://guidocioni.altervista.org/>
guidocioni.altervista.org<http://guidocioni.altervista.org/>
Le stazioni sono state riparate ed i dati vengono nuovamente aggiornati in tempo reale. Purtroppo a causa di numerosi malfunzionamenti i dati pluviometrici di Pisa e ...

On 12 Jan 2016, at 15:35, Michael Notaro <mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>> wrote:

Does anyone have a recommendation to speed up my code?
It's been running for a day now.  I put asterisks next to the real slow loop.
Basically, that part is converting a large array of hourly data into daily data.
Thanks, Michael

mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)

data=new((/141,217,20,12,744/),float) ; hourly data
data at _FillValue=1e+35
data=1e+35

cnt=0
do iyr=0,19
  do im=0,11
    prefix=(1980+iyr)+""+mns(im)
    b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"0100.nc","r") ; read hourly SRF data
    iy=b->iy
    jx=b->jx
    xlat=b->xlat
    xlon=b->xlon
    snow=b->snv ; liquid equiv of snow on ground
    dims=dimsizes(snow)
    nt=dims(0)
    data(:,:,iyr,im,0:nt-1)=snow(iy|:,jx|:,time|:)
    delete(snow)
    delete(b)
    delete(dims)
    cnt=cnt+1
  end do
end do
data at _FillValue=1e+20

daily=new((/141,217,20,12,31/),float) ; daily data per month
daily at _FillValue=1e+35
daily=1e+35

cnt=0
do id=0,30
  daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to daily data     ***** THIS PART IS SLOW *****
  cnt=cnt+24
end do

delete(data)

Michael Notaro
Associate Director
Nelson Institute Center for Climatic Research
University of Wisconsin-Madison
Phone: (608) 261-1503
Email: mnotaro at wisc.edu<mailto:mnotaro at wisc.edu>
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/e8adbc3f/attachment.html 