[ncl-talk] Slow code

Dennis Shea shea at ucar.edu
Tue Jan 12 13:47:16 MST 2016


At my request, Mike sent me two sample files. I will look when I get a
chance.

'My' strategy when dealing with multiple files is

[1]
Will use of 'addfiles' be advantageous for the problem.? if hourly arrays
are 20GB ... the answer is likely 'no'. Memory allocation, especially on
multi user systems) can be slow.

[2]
Most commonly

  [a] I will process one year at a time, converting year, month, day, hr to
(say) 'hours since ...'.
  [b] Compute the daily averages; write each year or month to netCDF with
'time' unlimited
  [c] use the 'ncks' or 'ncrcat' operators to combine the files if that is
desirable.

---
I am attaching the 'contrbuted.ncl' functions that are used to calculate
monthly or daily quantities: avg, sum, min, max



http://www.ncl.ucar.edu/Document/Functions/Contributed/calculate_daily_values.shtml
http://www.ncl.ucar.edu/Document/Functions/Contributed/calculate_monthly_values.shtml

D

On Tue, Jan 12, 2016 at 12:48 PM, Michael Notaro <mnotaro at wisc.edu> wrote:

> Thanks, Mary, for your further suggestions.
>
>
> After I got Alan's first email which helped me reassess my code,
>
> I modified my code to remove the year dimension from most variables
>
> to make them more manageable.  Now the code
>
> runs in 1 hour, rather than 1 day+.
>
>
> Michael
>
>
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/shea_util.ncl"
> begin
>
> mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
> ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)
>
> daily365=new((/365,20,141,217/),float) ; 365 day calendar data
> daily365 at _FillValue=1e+35
> daily365=1e+35
>
> do iyr=0,19
>
>   data=new((/141,217,12,744/),float) ; hourly data
>   data at _FillValue=1e+35
>   data=1e+35
>
>   cnt=0
>   do im=0,11
>     prefix=(1980+iyr)+""+mns(im)
>
> b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"
> 0100.nc","r") ; read hourly SRF data
>     iy=b->iy
>     jx=b->jx
>     xlat=b->xlat
>     xlon=b->xlon
>     snow=b->snv ; liquid equiv of snow on ground
>     dims=dimsizes(snow)
>     nt=dims(0)
>     data(:,:,im,0:nt-1)=snow(iy|:,jx|:,time|:)
>     delete(snow)
>     delete(b)
>     delete(dims)
>     cnt=cnt+1
>   end do
>   data at _FillValue=1e+20
>
>   daily=new((/141,217,12,31/),float) ; daily data per month
>   daily at _FillValue=1e+35
>   daily=1e+35
>
>   cnt=0
>   do id=0,30
>     daily(:,:,:,id)=dim_avg(data(:,:,:,cnt:cnt+23)) ; convert hourly to
> daily data
>     cnt=cnt+24
>   end do
>
>   delete(data)
>
>
>
>   cnt=0
>   do im=0,11
>     do id=0,ndays(im)-1
>       daily365(cnt,iyr,:,:)=daily(:,:,im,id) ; convert daily data per
> month to 365 day calendar
>       cnt=cnt+1
>     end do
>   end do
>
>   delete(daily)
>
> end do
>
> daily212=new((/19,212,141,217/),float) ; 212 day calendar data for Sep-Mar
> daily212 at _FillValue=1e+35
> daily212=1e+35
>
> do iyr=0,18
>   daily212(iyr,0:121,:,:)=daily365(243:364,iyr,:,:) ; retrieve Sep-Mar
>   daily212(iyr,122:211,:,:)=daily365(0:89,iyr+1,:,:)
> end do
>
> delete(daily365)
>
> year=ispan(0,18,1)
> year!0="year"
> year&year=year
>
> time=ispan(0,211,1)
> time!0="time"
> time&time=time
>
> daily212!0="year"
> daily212!1="time"
> daily212!2="iy"
> daily212!3="jx"
> daily212&year=year
> daily212&time=time
> daily212&iy=iy
> daily212&jx=jx
>
> daily212 at long_name = "liquid snow water on ground"
> daily212 at units = "kg m-2"
> daily212 at coordinates="xlat xlon"
> daily212 at grid_mapping = "rcm_map"
>
>
>
> system("rm save_daily212_actual_snv_access_late20_faster.nc")
> out=addfile("save_daily212_actual_snv_access_late20_faster.nc","c")
> out->daily212=daily212
> out->xlat=xlat
> out->xlon=xlon
> out->iy=iy
> out->jx=jx
>
>
>
>
>
> Michael Notaro
> Associate Director
> Nelson Institute Center for Climatic Research
> University of Wisconsin-Madison
> Phone: (608) 261-1503
> Email: mnotaro at wisc.edu
>
>
> ------------------------------
> *From:* Mary Haley <haley at ucar.edu>
> *Sent:* Tuesday, January 12, 2016 1:34 PM
> *To:* Alan Brammer
> *Cc:* Michael Notaro; ncl-talk at ucar.edu
>
> *Subject:* Re: [ncl-talk] Slow code
>
> Hi folks,
>
> These are all good suggestions.
>
> Another thing that is expensive in NCL is reordering arrays with syntax
> like:
>
> snow(iy|:,jx|:,time|:),
>
> NCL makes a copy of the array when you do this, and it has to swap all the
> dimensions every time in the loop.
>
> Isf reordering the array is absolutely necessary? I see that you are
> reordering and then calling "dim_avg_n". Since you are already using
> dim_avg_n, why not leave the array as is and just change the dimension you
> do the average on?
>
> --Mary
>
>
> On Tue, Jan 12, 2016 at 8:30 AM, Alan Brammer <abrammer at albany.edu> wrote:
>
>> Hi Michael,
>>
>>
>> I was going to suggest reshape that data array but it’s 20GB and is going
>> to be unnecessarily slow whatever.  Do you actually need to store all the
>> hourly data? the below edits suggest that you don’t.  The below uses less
>> than a 1GB of memory rather than 20+GB.
>>
>>  This is obviously untested so may need editing.
>> (requires 6.1.1 or newer. )
>>
>>
>> mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
>> ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)
>>
>> daily=new((/141,217,20,12,31/),float) ; hourly data
>> daily at _FillValue=1e+35
>> daily=1e+35
>>
>> cnt=0
>> do iyr=0,19
>>   do im=0,11
>>     prefix=(1980+iyr)+""+mns(im)
>>
>> b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"
>> 0100.nc","r") ; read hourly SRF data
>>     iy=b->iy
>>     jx=b->jx
>>     xlat=b->xlat   ; These aren’t doing anything?
>>     xlon=b->xlon ; These aren’t doing anything?
>>     snow =b->snv ; liquid equiv of snow on ground
>>     dims=dimsizes(snow)
>>     nt=dims(0)
>>
>>     snow4d := reshape( snow(iy|:,jx|:,time|:), (/dims(1), dims(2),
>> ndays(im), 24/) ) ; I assume snow is originally (time|:,iy|:,ix|:)
>>     daily(:,:,iyr,im,:ndays(im)-1)=dim_avg_n(snow4d, 3)
>>
>>     delete(snow)
>>     delete(b)
>>     delete(dims)
>>     cnt=cnt+1
>>   end do
>> end do
>>
>> daily at _FillValue=1e+20
>>
>>
>> Good luck,
>>
>>
>> Alan Brammer.
>>
>>
>>
>>
>> On 12 Jan 2016, at 10:00, Michael Notaro <mnotaro at wisc.edu> wrote:
>>
>> Thanks for your email.
>>
>> Actually, this is the main part slowing me down, not the top part of the
>> code with the addfiles.
>>
>> cnt=0
>> do id=0,30
>>   daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to
>> daily data     ***** THIS PART IS SLOW *****
>>   cnt=cnt+24
>> end do
>>
>>
>> Any way to perform this task quicker?
>>
>> Michael
>>
>>
>>
>>
>>
>> Michael Notaro
>> Associate Director
>> Nelson Institute Center for Climatic Research
>> University of Wisconsin-Madison
>> Phone: (608) 261-1503
>> Email: mnotaro at wisc.edu
>>
>>
>> ------------------------------
>> *From:* Guido Cioni <guidocioni at gmail.com>
>> *Sent:* Tuesday, January 12, 2016 8:57 AM
>> *To:* Michael Notaro
>> *Cc:* ncl-talk at ucar.edu
>> *Subject:* Re: [ncl-talk] Slow code
>>
>> Everyone here will tell you that using loops in NCL it’s not efficient :)
>> But from my experience I think that the main thing slowing you down is
>> that you are using addfile at every iteration.
>> Does creating a whole file and reading that in the beginning would change
>> what you are trying to compute?
>>
>> Guido Cioni
>> http://guidocioni.altervista.org
>> <http://guidocioni.altervista.org/>
>> Guido Cioni <http://guidocioni.altervista.org/>
>> guidocioni.altervista.org
>> Le stazioni sono state riparate ed i dati vengono nuovamente aggiornati
>> in tempo reale. Purtroppo a causa di numerosi malfunzionamenti i dati
>> pluviometrici di Pisa e ...
>>
>>
>> On 12 Jan 2016, at 15:35, Michael Notaro <mnotaro at wisc.edu> wrote:
>>
>> Does anyone have a recommendation to speed up my code?
>> It's been running for a day now.  I put asterisks next to the real slow
>> loop.
>> Basically, that part is converting a large array of hourly data into
>> daily data.
>> Thanks, Michael
>>
>>
>> mns=(/"01","02","03","04","05","06","07","08","09","10","11","12"/)
>> ndays=(/31,28,31,30,31,30,31,31,30,31,30,31/)
>>
>> data=new((/141,217,20,12,744/),float) ; hourly data
>> data at _FillValue=1e+35
>> data=1e+35
>>
>> cnt=0
>> do iyr=0,19
>>   do im=0,11
>>     prefix=(1980+iyr)+""+mns(im)
>>
>> b=addfile("/volumes/data1/yafang/Downscaling/ACCESS1-0/historical/output/ACCESS_SRF."+(1980+iyr)+""+mns(im)+"
>> 0100.nc","r") ; read hourly SRF data
>>     iy=b->iy
>>     jx=b->jx
>>     xlat=b->xlat
>>     xlon=b->xlon
>>     snow=b->snv ; liquid equiv of snow on ground
>>     dims=dimsizes(snow)
>>     nt=dims(0)
>>     data(:,:,iyr,im,0:nt-1)=snow(iy|:,jx|:,time|:)
>>     delete(snow)
>>     delete(b)
>>     delete(dims)
>>     cnt=cnt+1
>>   end do
>> end do
>> data at _FillValue=1e+20
>>
>> daily=new((/141,217,20,12,31/),float) ; daily data per month
>> daily at _FillValue=1e+35
>> daily=1e+35
>>
>> cnt=0
>> do id=0,30
>>   daily(:,:,:,:,id)=dim_avg(data(:,:,:,:,cnt:cnt+23)) ; convert hourly to
>> daily data     ***** THIS PART IS SLOW *****
>>   cnt=cnt+24
>> end do
>>
>> delete(data)
>>
>>
>>
>>
>> Michael Notaro
>> Associate Director
>> Nelson Institute Center for Climatic Research
>> University of Wisconsin-Madison
>> Phone: (608) 261-1503
>> Email: mnotaro at wisc.edu
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/389c0b06/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: calcMonthDay.ncl
Type: application/octet-stream
Size: 16077 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/389c0b06/attachment.obj 


More information about the ncl-talk mailing list