<div dir="ltr"><div><div><div><div><div><div><div>Hi Walter & Dennis,<br><br></div>Thank you so much for your both reply. I appreciate that you both take this issue seriously. Here are more info about my NCL code and SEVIRI HDF5 file.<br><br></div>(1 ). The NCL I am using is Version 6.3.0<br><br></div>(2 ) I tried to upload the data into FTP. Unfortunately, I failed and am not sure why. Since I can't upload the SEVIRI data into the FTP which Dennis provided. I uploaded the data into dropbox. You both can download from this link. <br><a href="https://www.dropbox.com/sh/cub21u3qxga2t5v/AAAjsa99xwG7cYsAcrJIntT3a?dl=0">https://www.dropbox.com/sh/cub21u3qxga2t5v/AAAjsa99xwG7cYsAcrJIntT3a?dl=0</a><br><br>Here are three file names<br>SEV_06-CLD-L2_2008-06-01T05-15-00_V1-03.hdf<br>SEV_06-CLD-L2_2008-06-01T05-30-00_V1-03.hdf<br>SEV_06-CLD-L2_2008-06-01T05-45-00_V1-03.hdf<br><br></div><div>Since the SEVIRI observation has one file per 15 min, and one hdf file is a little bit more than 100MB. So for daily data, the data size is about 15GB, which is pretty large. If I use "addfile" to load these files into the memory, will the memory be released after one loop to the next loop?<br><br></div><div>The data dimension is 3712*3712, and the data type is integer.<br><br></div>(3 ). Acutally in my code, I also also include a subroutine to plot the data out. I am not sure whether the slower process is related to the plot subroutine, which includes some basic resources.<br><br></div>(4). I will try these methods provided by you two and figure out which one is better for my code.<br><br></div>Thank you<br></div>Zhifeng<br><div><div><div><div><div><div><br><br><br></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 26, 2015 at 2:08 PM, Dennis Shea <span dir="ltr"><<a href="mailto:shea@ucar.edu" target="_blank">shea@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This type of behavior *may* be associated with a memory leak. However, more information is needed.<div><br></div><div>[0]</div><div>Please *always* include what version of NCL you are using. </div><div><br></div><div>[1] </div><div>As noted by Walter, multiple embedded do loops (you have 5 levels) in any interpreted language (NCL, Matlab, Python, R, ...) will lead to slow execution times. Again, as noted by Walter, you may have to 'rethink' your approach to obtaining the variable. </div><div><div><br></div><div>[2] </div><div>You have provided no indication of the format of the SEVERI files you are using.</div><div><br></div><div>The only SEVERI file in the NCL archive is a HDF5 file. It has no variable named '<span style="font-size:12.8000001907349px">Cloud_Optical_Thickness_16'. You may point ncl-talk to a WWW location from which some sample SEVERI files of the the type you are using may be obtained. Or you can ftp some files to:</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px"> ftp <a href="http://ftp.cgd.ucar.edu" target="_blank">ftp.cgd.ucar.edu</a></span></div><div><span style="font-size:12.8000001907349px"> anonymous</span></div><div><span style="font-size:12.8000001907349px"> email</span></div><div><span style="font-size:12.8000001907349px"> cd incoming</span></div><div><span style="font-size:12.8000001907349px"> put ... SEVERI_file_01...</span></div><div><span style="font-size:12.8000001907349px"> </span><span style="font-size:12.8000001907349px">put ... SEVERI_file_02...</span></div><div><span style="font-size:12.8000001907349px"> quit</span></div><div><br></div><div>[3]</div><div>As NCL gains more experience with handling HDF5, there have been *significant* improvements in handling HDF5 (H5) files over the last few NCL releases. See the following which document HDF5 improvements.</div><div><br></div><div> 6.3.1: <a href="http://www.ncl.ucar.edu/future_release.shtml" target="_blank">http://www.ncl.ucar.edu/future_release.shtml</a></div><div> Not yet released **but** a test binary is **available**</div><div><br></div><div> 6.3.0 and previous releases:</div><div> <a href="http://www.ncl.ucar.edu/prev_releases.shtml" target="_blank">http://www.ncl.ucar.edu/prev_releases.shtml</a></div><div><br></div><div>In particular, 6.3.1 seems to be fairly robust.</div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Fri, Jun 26, 2015 at 10:23 AM, Zhifeng Yang <span dir="ltr"><<a href="mailto:yangzf01@gmail.com" target="_blank">yangzf01@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div><div><div>Hi<br><br></div>I am trying to read SEVIRI data with a lot of variables and the dimension of each variable is 3712*3712. I know the data are pretty large. But the computer should read them smoothly. Since the memory that I specified is about 50GB. Unfortunately, the code is becoming slower and slower while it do the time loop. Here is a sample of my code.<br><br>; SET UP THE START TIME AND END TIME<br> start_year = 2008<br> end_year = 2008<br> start_month= 6<br> end_month = 6<br> start_day = 1<br> start_hour = 0<br> end_hour = 23<br> start_min = 0<br> end_min = 45<br> min_stride = 15<br> start_ind_lat = 1400<br> end_ind_lat = 3000<br> start_ind_lon = 1100<br> end_ind_lon = 2600<br><br>; DO YEAR LOOP<br> do iyear = start_year, end_year<br><br>; DO MONTH LOOP<br> do imonth = start_month, end_month<br><br>; CALCULATE THE NUMBER OF DAYS IN THIS MONTH<br> nday_month = days_in_month(iyear, imonth)<br>; DO DAY LOOP<br> do iday = start_day, 10;nday_month<br>; DO HOUR LOOP<br> do ihour = start_hour, end_hour<br>; DO MINUTE LOOP<br> do imin = start_min, end_min, min_stride<br>; READ VARIABLES FROM HDF FILE<br> a = addfile(dir + siyear + "/" + symd1 + "/" + filename, "r")<br> lat = (/a->MSG_Latitude(start_ind_lat:end_ind_lat, start_ind_lon:end_ind_lon)/)<br> lon = (/a->MSG_Longitude(start_ind_lat:end_ind_lat, start_ind_lon:end_ind_lon)/)<br> Cloud_Optical_Thickness_16 = a->Cloud_Optical_Thickness_16(start_ind_lat:end_ind_lat, start_ind_lon:end_ind_lon)<br><br> end do ;imin<br> end do ;ihour<br> end do ;iday<br> end do ;imonth<br> end do ;iyear<br><br><br></div>Thank you<span><font color="#888888"><br></font></span></div><span><font color="#888888">Zhifeng<br></font></span></div>
<br></div></div><span class="">_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></span></blockquote></div><br></div>
</blockquote></div><br></div>