<div dir="ltr"><div><div><div><div><div>Under-the-hood, NCL is calling the standard C-interfaces to the netCDF library. <br><br></div>Your file is 48GB. Each variable is less but what size?<br><br></div>Are you in a multi-user environment? <br><br></div>As far as I know, operating systems do mange memory. If a user asks for 2gb, the system may grant only (say) 200MB; then it round robins to other users granting memory before coming back to you for another 200MB, etc. Lots of users can cause slownwss.<br><br></div>The downside is the memory is not contiguous. Rather, it is fragmented. This is not an NCL issue.<br><br></div>The 'relhum' function NCL uses is from a fortran based model post-processor. NCL was requested to use this for backward compatibility. It is a function that uses a table look-up method. While this function is fortran, i does one computation at a time. The actual looping is done by a C-driver. Certainly, this language mix inhibits optimization. Still it should be pretty quick.<br><div><br><div><br></div><div><font style="font-size:11px" face="Menlo"> data = addfile("./<a href="http://complete_remap.nc" target="_blank">complete_remap.nc</a>", "r")</font></div><div><font style="font-size:11px" face="Menlo"> <br><br></font></div><div><font style="font-size:11px" face="Menlo">; read all data<br><br></font></div><div><font style="font-size:11px" face="Menlo"> readStrt_p = get_cpu_time()<br></font></div><div><font style="font-size:11px" face="Menlo"> p = data->pres ; pressure [Pa]<br> </font>print("readStrt_: " + (<strong>get_cpu_time</strong>() - readStrt_p))<br></div><div> <br></div><div><font style="font-size:11px" face="Menlo"> t = data->temp ; temperature [K]</font></div><div><font style="font-size:11px" face="Menlo"> qv = data->qv ; qv [ kg/kg]</font></div><div><font style="font-size:11px" face="Menlo"> z = data->z_mc ; geopotential [m]</font></div><div><font style="font-size:11px" face="Menlo"> <br> </font>print("read All Variables: " + (<strong>get_cpu_time</strong>() - readStrt_p))<br><br></div><div> print("===")<br></div><div> printVarSummary(p)<br><div><font style="font-size:11px" face="Menlo"><br></font></div><div><font style="font-size:11px" face="Menlo">; computations ... all<br> </font><br></div><div><font style="font-size:11px" face="Menlo"> rhStrt = get_cpu_time()</font></div></div><div><font style="font-size:11px" face="Menlo"> rh=relhum(t, qv, p)<br></font></div><div><font style="font-size:11px" face="Menlo"> </font>print("rh: " + (<strong>get_cpu_time</strong>() - rhStrt))<br><br></div><div> tdStrt = <font style="font-size:11px" face="Menlo">get_cpu_time()</font></div><div><font style="font-size:11px" face="Menlo"> td=dewtemp_trh(t, rh)</font></div><div><font style="font-size:11px" face="Menlo"> </font>print("td: " + (<strong>get_cpu_time</strong>() - tdStrt))<br><br></div><div><font style="font-size:11px" face="Menlo"> print("COMPUTATION "+(get_cpu_time()-rhStrt)+"s”)<br></font></div><div><font style="font-size:11px" face="Menlo"> print("======================================================")<br><br></font></div><div>You could also use a much smaller array size just to see what happens<br><br><font style="font-size:11px" face="Menlo"> readStrt_p = get_cpu_time()<br></font><div><font style="font-size:11px" face="Menlo"> p = data->pres(0:3,:,:) ; pressure [Pa]<br> </font>print("readStrt_: " + (<strong>get_cpu_time</strong>() - readStrt_p))<br></div><div> <br></div><div><font style="font-size:11px" face="Menlo"> t = data->temp(0:3,:,:) ; temperature [K]</font></div><div><font style="font-size:11px" face="Menlo"> qv = data->qv(0:3,:,:) ; qv [ kg/kg]</font></div><div><font style="font-size:11px" face="Menlo"> z = data->z_mc(0:3,:,:) ; geopotential [m]</font></div><br></div> printVarSummary(p)</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 6, 2016 at 2:40 AM, Guido Cioni <span dir="ltr"><<a href="mailto:guidocioni@gmail.com" target="_blank">guidocioni@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Just forgot to add that the time dimension has “only” 73 steps. <span class="HOEnZb"><font color="#888888"><br><div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br>Guido Cioni</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://guidocioni.altervista.org" target="_blank">http://guidocioni.altervista.org</a> </div>
</div></font></span><div><div class="h5">
<br><div><blockquote type="cite"><div>On 06 Jun 2016, at 10:38, Guido Cioni <<a href="mailto:guidocioni@gmail.com" target="_blank">guidocioni@gmail.com</a>> wrote:</div><br><div><div style="word-wrap:break-word">Hi,<div>the question is very simple, and I believe to already have the answer but still, is worth trying.</div><div>When managing large files in NCL I always have to create new tricks in the debugging phase in order to avoid long waiting times. Today I was trying to read a dataset with 401x401x150 points (approx 48 GB):</div><div><br></div><div><div><font style="font-size:11px" face="Menlo"> data = addfile("./<a href="http://complete_remap.nc" target="_blank">complete_remap.nc</a>", "r")</font></div><div><font style="font-size:11px" face="Menlo"> </font></div><div><font style="font-size:11px" face="Menlo"> p = data->pres ; pressure [Pa]</font></div><div><font style="font-size:11px" face="Menlo"> t = data->temp ; temperature [K]</font></div><div><font style="font-size:11px" face="Menlo"> qv = data->qv ; qv [ kg/kg]</font></div><div><font style="font-size:11px" face="Menlo"> z = data->z_mc ; geopotential [m]</font></div><div><font style="font-size:11px" face="Menlo"> </font></div><div><font style="font-size:11px" face="Menlo"> print("FILEs READ in "+get_cpu_time()+"s")</font></div><div><font style="font-size:11px" face="Menlo"><br></font></div><div><font style="font-size:11px" face="Menlo"> rh=relhum(t, qv, p)</font></div><div><font style="font-size:11px" face="Menlo"> td=dewtemp_trh(t, rh)</font></div><div><font style="font-size:11px" face="Menlo"> </font></div><div><font style="font-size:11px" face="Menlo"> print("COMPUTATION "+get_cpu_time()+"s”)</font></div><div><br></div><div>and getting the following printout.</div><div><br></div><div><div><font style="font-size:11px" face="Menlo">(0)<span style="white-space:pre-wrap"> </span>FILEs READ in 47.4748s</font></div><div><font style="font-size:11px" face="Menlo">(0)<span style="white-space:pre-wrap"> </span>COMPUTATION 499.424s</font></div></div><div><br></div><div>Is there any way to speed up the process? I tried to use as few definition as possible and only pre-included functions. </div><div>Why is the computation part taking so long? Maybe it’s something that depends on the system RAM? </div><div>In the meantime the best workaround that I could think of consisted in subsetting a region in the previous data and testing the code only on that file.</div><div><br></div><div>Cheers</div><div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br>Guido Cioni</div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://guidocioni.altervista.org/" target="_blank">http://guidocioni.altervista.org</a> </div>
</div>
<br></div></div></div></blockquote></div><br></div></div></div><br>_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>