<div dir="ltr"><div>As noted by Rashed, the question is vague:</div><div><br></div><div><div>Not sure what you by "efficiently" , the size of your variable or what statistics are to be calculated.</div><div>Really, the issue is the size of your variable(s).<br></div><div><br></div><div> NCL uses the standard Unidata netCDF software [written in C]. As Unidata has stated, their software was designed for robustness not necessarily for efficiency. There will be 2847 netCDF file openings. I don't think any other language would do this more or less efficiently than NCL.<br></div><div><br></div><div>What sizes are "level,lat,lon"? What is the size of:</div><div><br></div><div> nday = 2847l ; the appended 'l' make it a long integer [same as<b> tolong</b>(2846)]<br> ntim = nday*24<br> klvl = <br> nlat = <br> mlon = <br></div><div> n = ntim*klvl*nlat*mlon ; number of elements<br></div><div> print(n)</div><div> <br><div> nbyte = 4 ; number of bytes per float number<br></div><div> N = n*nbyte</div><div> print(N) ; [minimum] memory needed in bytes<br></div></div><div><br></div><div>;============================</div><div>; *IF* you have the memory then the following approach should work.</div><div>; NOTE: NCL uses the C-language 'malloc' to allocate memory.</div><div>; <b>THERE IS NO MAGIC: </b>IF N IS VERY LARGE THIS WILL TAKE TIME!</div><div>; Maybe ... considerable time! <br></div><div>;=============================<br></div><div> diri = "./" ; input directory<br></div><div> fili = systemfunc("cd "+diri+" ; ls HUANG*nc")</div><div> print(fili) ; print al 2847 file names<br></div><div>;<br></div><div> f = addfiles(diri+fili, "r") ; open all 2847 files<br></div><div> wsp= f[:]->WSP ; input (mtim,klvl,nlat,mlon)</div><div> ; this will take time because memory must be allocated<br></div><div><br></div><div> wsp_avg = dim_avg_n_Wrap(wsp, 0) ; 0 is the 'time' dimension<br></div><div> printVarSummary(wsp_avg) ; (level,lat,lon)</div><div><br></div><div>... other statistics</div><div>....save to netCDF file<br></div><div><br></div><div>*******************************************************</div><div>If memory limitations prevent all hourly grids, you will have to process some grid subset then, later, combine the information using the CDO or NCO operators. You may have to experiment to determine the the subset sizes that can be accommodated by your local environment. t<br></div><div><br></div><div><div> fili = systemfunc("cd "+diri+" ; ls HUANG*nc")</div><div> print(fili) ; print al 2847 file names<br></div><div>;<br></div><div> f = addfiles(diri+fili, "r") ; open all 2847 files</div><div><br></div><div> latS = -20 ; example of subset<br></div><div> latN = 0</div><div> lonL = 0</div><div> lonR = 10</div><div> <br></div><div> wsp= f[:]->WSP(:,:,<b>{latS:latN},{</b><b>lonL:lonR}</b>) ; input (mtim,klvl,...,...)</div><div> ; this will take time because memory must be allocated<br></div><div><br></div><div> wsp_avg = dim_avg_n_Wrap(wsp, 0) ; 0 is the 'time' dimension<br></div><div> printVarSummary(wsp_avg) ; (level,{latS:latN},{lonL:lonR})</div><div><br></div><div>... other statistics</div><div>....save to netCDF file</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 22, 2020 at 8:02 AM Rashed Mahmood via ncl-talk <<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Huang,<br></div><div>This question is a bit vague, at least I am not what statistics you are trying to calculate. NCL can do array processing for certain things, however, for a better answer to your specific needs, you would need to be specific about what exactly you are trying to do? <br></div><div><br></div><div>Rashed<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 21, 2020 at 6:56 PM 时光足迹 via ncl-talk <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Hi there, <br></div><div> I have problems processing data with numerous files (~2GB each file) using NCL. <br></div><div> There are 10-year hourly 3-dimention meteological data in NetCDF format, which from 2010~2019, 2847 days in total. Data were stored daily (so there are 2847 files). <br></div><div> I want to caluclate some statistic varibles along all time in each grid. Take wsp(time|2847*24, level, lat, lon) for example, in the current method, do-loop on levels is used. Then <br></div><div>'wsp' will be processed along time. Finally I can get what I need ( wsp_process(level,lat,lon) ）.</div><div> However, it will be really ineffiency, and sometime it may meet error 'Segmentation fault (core dumped)'. <br></div><div> Is there any effiency way to meet my needs? <br></div><div>Best regards,</div><div>HUANG<br> </div>_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a></blockquote></div>
_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a></blockquote></div>