[ncl-talk] The slow down when addfile is used in do loop

[Kyle Griffin]

Deleting "dat", which is your file pointer, does exactly what it should do
- deletes the variable that points to your file. This is, at most, a few
dozen bytes. Compared to entire globes worth of temperature data for 30
years, we're talking 10 orders of magnitude too small to make a difference.
The only way to reduce your memory usage is to *read less data* - there is
no other way to speed this up, unless you have another machine. This is not
an issue with NCL, but rather with how you are designing your code. As the
scientist here, it is up to YOU to figure out how much data you have, know
if your machine can handle it, and try and determine the best way to
process it on your way to performing your analysis. Unless you are
performing a function that uses all 30 years of data at once, I would
suggest reading in the files one at a time (monthly it appears?) and
performing your calculations on that, then writing/plotting the output as
you go. This should make things much faster. If you do need all 30 years of
data at once, consider how you need it. Maybe you plan on spatially
averaging the data into daily points? If so, you can do that as you read it
in.

Basically, try and reduce the amount of unnecessary data that is sitting
around at any one point in time.

----------------------------------------
Kyle S. Griffin
Department of Atmospheric and Oceanic Sciences
University of Wisconsin - Madison
Room 1421
1225 W Dayton St, Madison, WI 53706
Email: ksgriffin2 at wisc.edu

On Fri, Sep 12, 2014 at 5:19 PM, <rupingmo at gmail.com> wrote:

> Hi Jon,
>
> I do have the latest version of NCL (v6.2.1). The code is running fast to
> the point where the previous run was terminated, then begins to slow down.
> The delete function (in my case, delete(dat)) at the end of each iteration
> doesn't seem to help.
>
> Ruping
> Sent from my BlackBerry device on the Rogers Wireless Network
> ------------------------------
> *From: * Jon Meyer <jonathan.meyer at aggiemail.usu.edu>
> *Date: *Fri, 12 Sep 2014 16:03:19 -0600
> *To: *Ruping Mo<rupingmo at gmail.com>; ncl-talk at ucar.edu<ncl-talk at ucar.edu>
> *Subject: *Re: [ncl-talk] The slow down when addfile is used in do loop
>
> What version of NCL are you using? Version 6.2 fixed a minor memory leak
> associated when .grb files are opened. When numerous .grb files are opened
> sequentially, the small memory leak can cause your memory footprint to grow
> with time...even if you are using the 'delete' function after each loop
> iteration.
>
> If your code is running fast at first, using the delete function at the
> end of each iteration will likely solve your issue.
>
> Jon
>
> On Fri, Sep 12, 2014 at 2:31 PM, Ruping Mo <rupingmo at gmail.com> wrote:
>
>> Hi NCLers,
>>
>> I have an archive of large grib files. Each grib file contains a month of
>> multiple daily global variables. I use the following script to extract a
>> small part of one variable. The script, which uses addfile in a large loop,
>> runs very fast for the first few months, then begins to slow down
>> substantially. I let it run for a few years, then terminate the job
>> (Ctrl-Z) and re-run the same script. Surprisingly, the new job runs very
>> fast to the month when the previous job was terminated. I wonder why it
>> behaves this way, and what I can do to avoid the slow down problem.
>>
>> Have a nice day!
>>
>> Ruping
>>
>> The script:
>>
>> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>> begin
>>  nday = 12784     ; number of days from Jan 1979 to Dec 2013
>>  temp = new((/nday, 9, 9/), float)
>> ;;;;
>>  t0 = 0
>>  do yr = 1979, 2013
>>    syr = tostring(yr)
>>    do mo = 1, 12
>>     smo = tostring(mo)
>>     print(syr + "-" + smo)
>>     fname = syr + "/" + syr + "-" + smo + ".grib"
>>     dat = addfile(fname, "r")
>>     mday = getfilevardimsizes(dat, "initial_time0") - 1
>>     i1 = i0 + mday
>>     temp(i0:i1, :, :) = (/ dat->2T_GDS0_SFC(:, 80:88, 146:154) /)
>> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; 2T_GDS0_SFC(mday, 241, 480)
>>     i0 = i1 + 1
>>    end do
>>  end do
>> end
>>
>> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20140912/9b3755f2/attachment.html