<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Loops are inherently slow in interpreted languages such as NCL. You
should reduce your loops to as few as possible. Here is a code
snippet I use in a lot of my code:<br>
<br>
<tt>; convert times into epoch-seconds</tt><tt><br>
</tt><tt>startTime = cd_inv_calendar( startYear, startMonth,
startDay, startHour, 0, 0, "seconds since 1970-1-1 00:00:00", 0 )</tt><tt><br>
</tt><tt>endTime = cd_inv_calendar( endYear, endMonth, endDay,
endHour, 0, 0, "seconds since 1970-1-1 00:00:00", 0 )</tt><tt><br>
</tt><tt>frequency = frequencyHr * 3600;</tt><tt><br>
</tt><tt><br>
</tt><tt>; build array with all of the initialization times</tt><tt><br>
</tt><tt>nTimes = doubletointeger( (endTime - startTime) / </tt><tt><tt>frequency
</tt>) + 1</tt><tt><br>
</tt><tt>times = fspan( startTime, endTime, nTimes )</tt><tt><br>
</tt><tt>copy_VarAtts( startTime, </tt><tt><tt>times </tt>)</tt><br>
<br>
If you need all of the data at the end (or otherwise can process all
the data at once), you can then use times to build an array of
filenames using cd_<tt>string</tt> and use <tt>addfiles</tt> to
read them all at once:<br>
<br>
<tt>yyyymmdd = cd_string( times, "%Y%N%D")</tt><tt><br>
</tt><tt>filenames = dir + "/" + yyyymmdd + ".hdf"</tt><tt><br>
</tt><tt>infiles = addfiles( filenames, "r" )</tt><tt><br>
</tt><tt>ListSetType(infiles, "join")</tt><tt><br>
</tt><tt>variable = infiles[:]->$variableName$</tt><tt>(:,
start_ind_lat:end_ind_lat, start_ind_lon:end_ind_lon)<br>
variable!0 = "time"<br>
variable&valid_time = times<br>
</tt><br>
If you need to do separate processing for each time, you've still
reduced everything down to a single loop instead of many:<br>
<br>
<tt>do t=0, nTimes-1, 1</tt><tt><br>
</tt><tt> time = times(t)</tt><br>
<tt><tt> yyyymmdd = cd_string( time, "%Y%N%D")</tt><tt><br>
</tt><tt> filename = dir + "/" + yyyymmdd + ".nc"<br>
infile = addfile( filename, "r" )<br>
</tt><tt>
</tt> variable = infile->$variableName$(</tt><tt>start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)</tt><tt><br>
; Do stuff to variable<br>
delete(variable)<br>
</tt><tt>end do</tt><br>
<br>
Even if you have to do separate processing, it may even be more
efficient to read all the data at once anyway and step through if
you have the memory:<br>
<br>
<tt>do t=0, nTimes-1, 1</tt><tt><br>
</tt><tt> </tt><tt>variable_temp = variable(t,:,:)</tt><tt><br>
; Do stuff to variable_temp<br>
delete(</tt><tt><tt>variable_temp</tt>)<br>
</tt><tt>end do</tt><br>
<br>
Also, if your lat and lon are the same at every time (likely), you
probably only need to read them in from the first file:<br>
<tt>lat = infiles[0]->lat</tt><tt>(start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)</tt><br>
<tt>lon = infiles[0]->lon</tt><tt>(start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)</tt><br>
<br>
or<br>
<br>
<tt>do t=0, nTimes-1, 1</tt><tt><br>
</tt><tt> time = times(t)<br>
...<br>
</tt>
<tt><tt> if( .not.isdefined("lat")</tt></tt><tt> ) then<br>
lat = infile->lat</tt><tt>(</tt><tt>start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)</tt><tt><br>
lon </tt><tt><tt>= infile->lon</tt><tt>(</tt><tt>start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)</tt><tt><br>
end if<br>
; Do stuff<br>
delete(variable)<br>
; DON'T delete lat or lon</tt><br>
</tt><tt>end do</tt><br>
<br>
Finally, when using loops you may want to delete any large variables
you are using on each loop (as I've done above). It shouldn't
matter, but I've found it does make a difference sometimes. And, if
you are making any outside function calls to FORTRAN/C++ subroutines
in your processing, make sure those functions are cleaning up after
themselves (particularly if you wrote them yourself).<br>
<br>
- Walter<br>
<br>
<div class="moz-cite-prefix">On 26-Jun-15 12:23, Zhifeng Yang wrote:<br>
</div>
<blockquote
cite="mid:CAF92WDMvG=Hb-zyE5Q+SqPOShxTz=Y0MCZdQcHJQGfcmUbXUzA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>Hi<br>
<br>
</div>
I am trying to read SEVIRI data with a lot of variables and
the dimension of each variable is 3712*3712. I know the data
are pretty large. But the computer should read them
smoothly. Since the memory that I specified is about 50GB.
Unfortunately, the code is becoming slower and slower while
it do the time loop. Here is a sample of my code.<br>
<br>
; SET UP THE START TIME AND END TIME<br>
start_year = 2008<br>
end_year = 2008<br>
start_month= 6<br>
end_month = 6<br>
start_day = 1<br>
start_hour = 0<br>
end_hour = 23<br>
start_min = 0<br>
end_min = 45<br>
min_stride = 15<br>
start_ind_lat = 1400<br>
end_ind_lat = 3000<br>
start_ind_lon = 1100<br>
end_ind_lon = 2600<br>
<br>
; DO YEAR LOOP<br>
do iyear = start_year, end_year<br>
<br>
; DO MONTH LOOP<br>
do imonth = start_month, end_month<br>
<br>
; CALCULATE THE NUMBER OF DAYS IN THIS MONTH<br>
nday_month = days_in_month(iyear, imonth)<br>
; DO DAY LOOP<br>
do iday = start_day, 10;nday_month<br>
; DO HOUR LOOP<br>
do ihour = start_hour, end_hour<br>
; DO MINUTE LOOP<br>
do imin = start_min, end_min, min_stride<br>
; READ VARIABLES FROM HDF FILE<br>
a = addfile(dir + siyear + "/" +
symd1 + "/" + filename, "r")<br>
lat =
(/a->MSG_Latitude(start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)/)<br>
lon =
(/a->MSG_Longitude(start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)/)<br>
Cloud_Optical_Thickness_16 =
a->Cloud_Optical_Thickness_16(start_ind_lat:end_ind_lat,
start_ind_lon:end_ind_lon)<br>
<br>
end do ;imin<br>
end do ;ihour<br>
end do ;iday<br>
end do ;imonth<br>
end do ;iyear<br>
<br>
<br>
</div>
Thank you<br>
</div>
Zhifeng<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
ncl-talk mailing list
<a class="moz-txt-link-abbreviated" href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a>
List instructions, subscriber options, unsubscribe:
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Walter Kolczynski, Jr.
Global Ensemble Team
NOAA/NWS/NCEP/EMC (via I.M. Systems Group)
(301) 683-3781</pre>
</body>
</html>