<div dir="ltr"><div dir="ltr"><div>Please CC the mailing list on your replies.</div><div><br></div>Serializing means processing in chunks, which you are already doing. I did not mean anything more than that. If you are getting memory errors, then perhaps your chunks are too large.<div><br></div><div>It is also possible that something in your code is trying to read the entire data set, and you don't realize it. For example, each of these statements seems innocent, but each one tries to read an entire array into memory before producing a very simple result. These will both cause memory error or slow performance with a very large file:</div><div><br></div><div> printVarSummary (f->x)</div><div> dims = dimsizes (f->x)</div><div><br></div><div>If that does not help solve the problem, then please show the complete NCL statement that gets the memory error, and the complete error message. Also show the dimensions of any arrays used in that statement.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 8, 2020 at 7:01 PM STEFAN RAHIMI-ESFARJANI <<a href="mailto:s.rahimi@ucla.edu">s.rahimi@ucla.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks for this,<div><br></div><div>3 TB is not all to be in memory simultaneously. I have set up the code to read "chunks" of data at a time, average them, and then move on.</div><div><br></div><div>Any specific recommendations are appreciated.... I'm not sure what you mean by serializing the code...</div><div><br></div><div>Thanks again,</div><div>-Stefan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 8, 2020 at 6:31 PM Dave Allured - NOAA Affiliate via ncl-talk <<a href="mailto:ncl-talk@mailman.ucar.edu" target="_blank">ncl-talk@mailman.ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>3 Tb is a very large data set, difficult to contain all in memory at the same time. I recommend using serial methods to perform your analysis along one or two dimensions, one step at a time. You can implement serial methods in NCL or any other scientific programming language. In NCL, you can use the full complement of library functions to assist your serial methods.</div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 8, 2020 at 12:08 PM Chathurika via ncl-talk <<a href="mailto:ncl-talk@mailman.ucar.edu" target="_blank">ncl-talk@mailman.ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
<p>
<span style="vertical-align:inherit"><span style="vertical-align:inherit"><span style="vertical-align:inherit"><span style="vertical-align:inherit">Hello,</span></span></span></span>
</p>
<p>
<span style="vertical-align:inherit"><span style="vertical-align:inherit"><br>
</span></span>
</p>
<p>
<span style="vertical-align:inherit"><span style="vertical-align:inherit"><span style="vertical-align:inherit"><span style="vertical-align:inherit">I want to handle large data set (more than 3TB).However, I am getting the error number 12 due to the failure of memory allocation. I read lot of Q&A for this issue, however there is no proper answer. I have to analyze this large data set. So please let me know how can I solve this memory allocation issue. I hope there is someone who already has a solution for this.</span></span></span></span></p>
<p>
<br>
</p>
<p>
<span style="vertical-align:inherit"><span style="vertical-align:inherit">Thank you so much and best regards</span></span></p>
<span>
<hr align="left" style="margin:0.5em 0px;width:10em;height:1px;background-color:rgb(153,153,153);border:none">
<blockquote style="margin:0px 0px 0px 40px;padding:0px;font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:small;font-variant-ligatures:normal;border:none"><span style="font-size:x-small;font-style:italic">Wickramage Chathurika Hemamali</span><br></blockquote><blockquote style="margin:0px 0px 0px 40px;padding:0px;font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-variant-ligatures:normal;border:none"><div>
<span style="font-style:italic"><span style="font-size:x-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">Msc in Physical Oceanography</span></span></span></span>
</div>
<div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2251">
<span style="font-style:italic"><span style="font-size:x-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">State Key Laboratory of Tropical Oceanography</span></span></span></span>
</div>
</div>
<div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2252">
<span style="font-style:italic"><span style="font-size:x-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">South China Sea Institute of Oceanology</span></span></span></span>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2252">
<span style="font-style:italic"><span style="font-size:x-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">University of Chinese Academy of Science</span></span></span></span>
</div>
</div>
<div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264">
<span style="font-style:italic"><span style="font-size:x-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">China</span></span></span></span>
</div>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<span style="font-style:italic;font-size:medium"><br>
</span>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<i><span style="font-size:xx-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">Specialized in Oceanography and Marine Geology (Bachelor)</span></span></span></i>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<i><span style="font-size:xx-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">University of Ruhuna</span></span></span></i>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<i><span style="font-size:xx-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">Matara</span></span></span></i>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<i><span style="font-size:xx-small"><span style="vertical-align:inherit"><span style="vertical-align:inherit">Sri Lanka</span></span></span></i>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<span style="font-style:italic;font-size:medium"><br>
</span>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<span style="font-style:italic;font-size:medium"><span style="vertical-align:inherit"><span style="vertical-align:inherit">Email: </span></span><a><span style="vertical-align:inherit"><span style="vertical-align:inherit">wickramagechathurika@rocketmail.com</span></span></a></span>
</div>
<div id="gmail-m_-4434947445412011471gmail-m_6496209199421152357gmail-m_239189906115319016ydpc6aa4ef1yui_3_16_0_ym19_1_1496286059659_2264" style="font-size:small">
<span style="font-style:italic;font-size:medium"><a><span style="vertical-align:inherit"><span style="vertical-align:inherit">chatu@scsio.ac.cn</span></span></a></span></div></blockquote></span></blockquote></div></div></blockquote></div>
</blockquote></div></div>