<div dir="ltr"><div class="gmail_default" style="font-size:small">Ehsan,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I had no trouble running your script with NCL V6.4.0. What version of NCL are you using? You can type "ncl -V" for this information.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">As I said in an earlier email, you can make the writing of the ASCII files faster if you don't use a double do loop. For example, this code:</div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> do i=0, dimsizes(lat_NW)-1</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> do j=0, dimsizes(time)-1</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> write_table("GPM_R24_Interpolated_slow.txt","a",[/time(j),lat_NW(i),lon_NW(i),GPM_fod(j,i)/], \</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> " %8i %7.4f %7.4f %5.1f")</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> end do</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> end do</font></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">can be replaced with:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default"><div class="gmail_default"><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> ntim = dimsizes(time)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> nlat = dimsizes(lat_NW)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> dims = (/nlat,ntim/)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> time1d = ndtooned(conform_dims(dims,time,1))</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> lat1d = ndtooned(conform_dims(dims,lat_NW,0))</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> lon1d = ndtooned(conform_dims(dims,lon_NW,0))</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> write_table("GPM_R24_Interpolated_fast.txt","a",[/time1d,lat1d,lon1d,transpose(GPM_fod)/], \</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> " %8i %7.4f %7.4f %5.1f")</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><br></font></div>Note that because you are writing the time dimension out first and then the lat/lon dimension, I had to transpose the GPM_fod array before writing it.</div><div class="gmail_default"><br>Also, the faster way does require the use of more memory, because I'm now creating extra arrays to hold all the time, lat, and lon arrays in a long 1D arrays.</div><div class="gmail_default"><br></div><div class="gmail_default">However, the speed up for writing the two files without do loops was significant, mainly for the first double do loop.</div><div class="gmail_default"><br></div>Original method with double do loops:</div><div class="gmail_default"><br></div><div class="gmail_default">
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-stretch:normal;font-size:14px;line-height:normal;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace" style=""><span class="gmail-Apple-tab-span" style="font-weight:normal;white-space:pre"> </span>Elapsed time for do loop = <b>299.03</b> CPU seconds</font></span></p></div></div><div class="gmail_default"><div class="gmail_default"><br>New method with no do loops:<br></div><div class="gmail_default"><br></div>
<p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-stretch:normal;font-size:14px;line-height:normal;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace" style=""><span class="gmail-Apple-tab-span" style="font-weight:normal;white-space:pre"> </span>Elapsed time for do loop = <b>0.702266</b> CPU seconds</font></span></p><p class="gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Monaco;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p>I've attached a modified version of your main.ncl script so you can see what I changed. Note that I changed the named of the output file names to have either the word "slow" or "fast" as part of them, so you can compare the two files. See the FASTER_WAY variable which I added to allow you to switch between the two methods. You probably want to remove these "if" statements once you verify the code is working the way you expect. </div><div class="gmail_default"><br></div><div class="gmail_default">Please try this new script to see if it fixes your issue.<br><br>--Mary</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 20, 2017 at 12:46 PM, Ehsan Taghizadeh via ncl-talk <span dir="ltr"><<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:bookman old style,new york,times,serif;font-size:16px"><div></div>
<div><br></div><div>Dear Mary</div><div>Thank you so much for your help. As you suggested I attached input files (GPM_IMERGV04DL_20170401_<wbr>20170430.rar) and script file (GPM_Interpolating.rar). The script works well until first, but it goes slow gradually and finally it killed.</div><div>About your following comment:</div><span class=""><div> <span style="font-family:monospace,monospace"> <span style="background-color:rgb(253,248,105)">write_table("file2.txt","w",<wbr>[/j,x,i,y,k/], "string_%03i %8.2f %4.0i %8.1f string_%03i")</span></span></div></span><div><span style="font-family:monospace,monospace"><span style="color:rgb(0,0,0);font-family:"bookman old style","new york",times,serif;font-size:16px">About I think it isn't possible for me, because of different indexes of variables:</span><br></span></div><div><span style="font-family:monospace,monospace"> write_table("GPM_R24_Grid.txt"<wbr>,"a",[/GPM_fnamesc</span><span style="font-family:monospace,monospace;background-color:rgb(228,172,100)">(i)</span><span style="font-family:monospace,monospace">,lat_GPM</span><span style="font-family:monospace,monospace;background-color:rgb(228,172,100)">(<wbr>j)</span><span style="font-family:monospace,monospace">,lon_GPM</span><span style="font-family:monospace,monospace;background-color:rgb(228,172,100)">(k)</span><span style="font-family:monospace,monospace">, R24h_GPM</span><span style="font-family:monospace,monospace;background-color:rgb(228,172,100)">(k,j)</span><span style="font-family:monospace,monospace">/], " %s %7.4f %7.4f %5.1f").</span><br></div><div><br></div><div>Sincerely</div><span class="HOEnZb"><font color="#888888"><div>Ehsan Taghizadeh</div></font></span><div><div class="h5">
<div id="m_-8565557200990124277yahoo_quoted_4263530150" class="m_-8565557200990124277yahoo_quoted">
<div style="font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;font-size:13px;color:#26282a">
<div>
On Tuesday, December 19, 2017, 8:10:12 PM GMT+3:30, Mary Haley <<a href="mailto:haley@ucar.edu" target="_blank">haley@ucar.edu</a>> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="m_-8565557200990124277yiv7470519977"><div><div dir="ltr"><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small">Dear Ehsan,</div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small"><br clear="none"></div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small">If you are asking us to look at your script and improve it for you, it helps to provide more details about what you expect the script to do and where it appears to be failing.</div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small"><br clear="none"></div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small">You included the script and some of the files, but not the NetCDF files, so there's no way for us to actually run the script to see what the problem is.</div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small"><br clear="none"></div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small">Did you try debugging your script to see where it might be failing? It would help to add more print statements, especially at the top of the do loops, so you can see how many iterations of the loops it completes before it fails.</div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small"><br clear="none"></div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small;display:inline">Nested do loops shouldn't cause your script to fail, but they could cause your script to be much slower than needed.</div><div><br clear="none"></div><div>I see that you are trying to write data to a file called "GPM_R24_Grid.txt" and that you have a double do loop<div class="m_-8565557200990124277yiv7470519977gmail_default" style="display:inline"> for writing individual rows to the file. You can write more than one row of data to the file without having to loop across each value, and this could greatly improve the speed of your script. </div></div><div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="display:inline"><br clear="none"></div></div><div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="display:inline">Here's a very simple example:</div></div><div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="display:inline"><br clear="none"></div></div><div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="display:inline"><div><font face="monospace, monospace">;---Generate some dummy integer and float data.</font></div><div><font face="monospace, monospace"> npts = 100</font></div><div><font face="monospace, monospace"> i = ispan(1,npts,1)</font></div><div><font face="monospace, monospace"> j = generate_unique_indices(npts)</font></div><div><font face="monospace, monospace"> k = generate_unique_indices(npts)</font></div><div><font face="monospace, monospace"> x = random_uniform(-10,10,npts)</font></div><div><font face="monospace, monospace"> y = random_uniform(0,1000.,npts)</font></div><div><font face="monospace, monospace"><br clear="none"></font></div><div><font face="monospace, monospace"> write_table("file2.txt","w",[/<wbr>j,x,i,y,k/], "string_%03i %8.2f %4.0i %8.1f string_%03i")</font></div><div><br clear="none"></div><div>For more examples of using write_table to write multiple lines of text to an ASCII file, see this examples page:</div><div><br clear="none"></div><div><a rel="nofollow" shape="rect" href="http://www.ncl.ucar.edu/Applications/write_csv.shtml" target="_blank">http://www.ncl.ucar.edu/<wbr>Applications/write_csv.shtml</a></div><div><br clear="none"></div><div>This page is for writing CSV files, but these examples can be used for any kind of ASCII file.</div><div><br clear="none"></div><div>If you continue to have problems with your script, please respond back to <a rel="nofollow" shape="rect" href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a> and provide more information, like where you think the script is failing, and what you are trying to accomplish with it.</div><div><br clear="none"></div><div>Thanks,</div><div><br clear="none"></div><div>--Mary</div><div><br clear="none"></div></div></div><div><br clear="none"></div><div><div><br clear="none"></div><div><br clear="none"><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small"><br clear="none"></div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small"><br clear="none"></div><div class="m_-8565557200990124277yiv7470519977gmail_default" style="font-size:small"><br clear="none"></div></div></div></div><div class="m_-8565557200990124277yiv7470519977gmail_extra"><br clear="none"><div class="m_-8565557200990124277yiv7470519977gmail_quote">On Tue, Dec 19, 2017 at 1:56 AM, Ehsan Taghizadeh via ncl-talk <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>></span> wrote:<br clear="none"><blockquote class="m_-8565557200990124277yiv7470519977gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-8565557200990124277yiv7470519977yqt4394761973" id="m_-8565557200990124277yiv7470519977yqt59517"><div><div style="font-family:bookman old style,new york,times,serif;font-size:16px"><div>Hi,</div><div>I try to run attached script (<a rel="nofollow" shape="rect" href="http://main.nc" target="_blank">main.nc</a>), however each time it counters killed, without any error.</div><div>I've searched google for that and it seems because of nested loops. Is it true? Could you please help to make this script more efficient? Also I attached some part of printVarSummary.</div><div><br clear="none"></div><div><i>Sincerely</i></div><span class="m_-8565557200990124277yiv7470519977HOEnZb"><font color="#888888"></font></span><div><i>Ehsan</i></div></div></div></div><br clear="none">______________________________ _________________<br clear="none">
ncl-talk mailing list<br clear="none">
<a rel="nofollow" shape="rect" href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br clear="none">
List instructions, subscriber options, unsubscribe:<br clear="none">
<a rel="nofollow" shape="rect" href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/ mailman/listinfo/ncl-talk</a><br clear="none">
<br clear="none"></blockquote></div><br clear="none"></div></div></div></div>
</div>
</div></div></div></div></div><br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>