[ncl-talk] Running the NCL code becomes slower

[Zhifeng Yang]

Hi Walter & Dennis,

Thank you so much for your both reply. I appreciate that you both take this
issue seriously. Here are more info about my NCL code and SEVIRI HDF5 file.

(1 ). The NCL I am using is Version 6.3.0

(2 ) I tried to upload the data into FTP. Unfortunately, I failed and am
not sure why. Since I can't upload the SEVIRI data into the FTP which
Dennis provided. I uploaded the data into dropbox. You both can download
from this link.
https://www.dropbox.com/sh/cub21u3qxga2t5v/AAAjsa99xwG7cYsAcrJIntT3a?dl=0

Here are three file names
SEV_06-CLD-L2_2008-06-01T05-15-00_V1-03.hdf
SEV_06-CLD-L2_2008-06-01T05-30-00_V1-03.hdf
SEV_06-CLD-L2_2008-06-01T05-45-00_V1-03.hdf

Since the SEVIRI observation has one file per 15 min, and one hdf file is a
little bit more than 100MB. So for daily data, the data size is about 15GB,
which is pretty large. If I use "addfile" to load these files into the
memory, will the memory be released after one loop to the next loop?

The data dimension is 3712*3712, and the data type is integer.

(3 ). Acutally in my code, I also also include a subroutine to plot the
data out. I am not sure whether the slower process is related to the plot
subroutine, which includes some basic resources.

(4). I will try these methods provided by you two and figure out which one
is better for my code.

Thank you
Zhifeng




On Fri, Jun 26, 2015 at 2:08 PM, Dennis Shea <shea at ucar.edu> wrote:

> This type of behavior *may* be  associated with a memory leak. However,
> more information is needed.
>
> [0]
> Please *always* include what version of NCL you are using.
>
> [1]
> As noted by Walter, multiple embedded do loops (you have 5 levels) in any
> interpreted language  (NCL, Matlab, Python, R, ...) will lead to slow
> execution times.  Again, as noted by Walter, you may have to 'rethink' your
> approach to obtaining the variable.
>
> [2]
> You have provided no indication of the format of the SEVERI files you are
> using.
>
> The only SEVERI file in the NCL archive is a HDF5 file.  It has no
> variable named 'Cloud_Optical_Thickness_16'. You may point ncl-talk to a
> WWW location from which some sample SEVERI files of the the type you are
> using may be obtained. Or you can ftp some files to:
>
>    ftp ftp.cgd.ucar.edu
>    anonymous
>    email
>    cd incoming
>    put ... SEVERI_file_01...
>    put ... SEVERI_file_02...
>    quit
>
> [3]
> As NCL gains more experience with handling HDF5, there have been
> *significant* improvements in handling HDF5 (H5) files over the last few
> NCL releases. See the following which document HDF5 improvements.
>
>     6.3.1:    http://www.ncl.ucar.edu/future_release.shtml
>                  Not yet released **but** a test binary is **available**
>
>     6.3.0 and previous releases:
>                  http://www.ncl.ucar.edu/prev_releases.shtml
>
> In particular, 6.3.1 seems to be fairly robust.
>
> On Fri, Jun 26, 2015 at 10:23 AM, Zhifeng Yang <yangzf01 at gmail.com> wrote:
>
>> Hi
>>
>> I am trying to read SEVIRI data with a lot of variables and the dimension
>> of each variable is 3712*3712. I know the data are pretty large. But the
>> computer should read them smoothly. Since the memory that I specified is
>> about 50GB. Unfortunately, the code is becoming slower and slower while it
>> do the time loop. Here is a sample of my code.
>>
>> ;  SET UP THE START TIME AND END TIME
>>    start_year = 2008
>>    end_year   = 2008
>>    start_month= 6
>>    end_month  = 6
>>    start_day  = 1
>>    start_hour = 0
>>    end_hour   = 23
>>    start_min  = 0
>>    end_min    = 45
>>    min_stride = 15
>>    start_ind_lat = 1400
>>    end_ind_lat   = 3000
>>    start_ind_lon = 1100
>>    end_ind_lon   = 2600
>>
>> ;  DO YEAR LOOP
>>    do iyear = start_year, end_year
>>
>> ;  DO MONTH LOOP
>>       do imonth = start_month, end_month
>>
>> ;  CALCULATE THE NUMBER OF DAYS IN THIS MONTH
>>          nday_month = days_in_month(iyear, imonth)
>> ;  DO DAY LOOP
>>          do iday = start_day, 10;nday_month
>> ;  DO HOUR LOOP
>>             do ihour = start_hour, end_hour
>> ;  DO MINUTE LOOP
>>                do imin = start_min, end_min, min_stride
>> ;  READ VARIABLES FROM HDF FILE
>>                      a     = addfile(dir + siyear + "/" + symd1 + "/" +
>> filename, "r")
>>                      lat   = (/a->MSG_Latitude(start_ind_lat:end_ind_lat,
>> start_ind_lon:end_ind_lon)/)
>>                      lon   =
>> (/a->MSG_Longitude(start_ind_lat:end_ind_lat, start_ind_lon:end_ind_lon)/)
>>                      Cloud_Optical_Thickness_16 =
>> a->Cloud_Optical_Thickness_16(start_ind_lat:end_ind_lat,
>> start_ind_lon:end_ind_lon)
>>
>>                end do ;imin
>>             end do ;ihour
>>          end do ;iday
>>       end do ;imonth
>>    end do ;iyear
>>
>>
>> Thank you
>> Zhifeng
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150626/758cec5d/attachment.html