<div dir="ltr"><div>[1] Not many servers allow or have available 80+GB of memory. You are lucky! :-)<br></div><div><br></div><div>[2] <br></div><div>You are using NCL's 'simple but, not necessarily, efficient' method to create netCDF.</div><div><br></div><div><a href="https://www.ncl.ucar.edu/Applications/write_netcdf.shtml"><b>https://www.ncl.ucar.edu/Applications/write_netcdf.shtml</b></a></div><div><br></div><div>This method takes advantage of the fact that NCL variable structures [objects] are based upon the netCDF variable model. Under the hood, NCL makes the appropriate calls to the C netCDF-library functions. For "small" files even with many variables, the method is 'efficient-enough'. The simplicity is VERY NICE. I would speculate the 95+% of netCDF are written via this method.<br></div><div><br></div><div>Still, because of the way netCDF files are structured there is a 'lot' of overhead runder-the-hood. Your data has 'only' 8 variables but they are LARGE. I am not sure how this affects memory usage.</div><div><br></div><div>[3] <br></div><div>My suggestion would be to use the netCDF operators [NCO] or Climate Data Operators [CDO].</div><div><br></div><div>NCO:</div><div>%> <b>ncrcat</b> -v dbz --file_format netcdf4 <span style="font-variant-ligatures:no-common-ligatures">/scratch/groups/oneillm/Nature_Runs/HNR1/domain4/RI_subsets/subset_*.nc" DBZ.nc<br></span></div><div><br></div><div>Perhaps, after all single variable files have been created:</div><div><br></div><div>%><b> ncks</b> DBZ.nc Q.nc ... MERGE.nc</div><div><br></div><div>Using the NCO or CDO operators is often recommended for these types of tasks. The are tuned to perform these tasks while NCL is a general purpose language.</div><div><br></div><div>Hope this helps</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 27, 2019 at 7:04 PM Prashanth Bhalachandran via ncl-talk <<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Dear NCL-ers, <div>I have a question regarding handling large amounts of data in NCL. </div><div><br></div><div>Currently, I have 41 Netcdf files that are each about 4.5 GBs. Each file contains eight variables that are <span style="color:rgb(180,36,25);font-family:Courier"> </span></div><div><span style="color:rgb(180,36,25);font-family:Courier">[Time | 10] x [bottom_top | 60] x [south_north | 480] x [west_east_stag | 481]</span></div><div><span style="color:rgb(180,36,25);font-family:Courier"><br></span></div><div><span>So far, I have managed to handle each of these files separately, but now I need to append them. I am using the following code but the data size is so large that the program is getting constantly killed even on servers (I tried requesting around 80 GB of memory). Is there a more efficient way to concatenate these files around the time dimension? I am attaching the code that I currently use below. To make the matters worse, I realized that some of my files are corrupt and need to be regenerated. However, I am unable to weed out which ones are corrupt without the below program running fully (that is currently difficult due to memory issues). </span></div><div><span><br></span></div><div><span>Can anyone help me out here, please? </span></div><div><font face="Courier"><span><br></span></font></div><div><font face="Courier"><span>Sai </span></font></div><div><br></div><div><br></div><div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures">all_files = systemfunc("ls /scratch/groups/oneillm/Nature_Runs/HNR1/domain4/RI_subsets/subset_*.nc")</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> setfileoption("nc","Format","NetCDF4")</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> fall=addfiles(all_files, "r")</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ListSetType(fall, "cat")</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier;min-height:22px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier;min-height:22px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> u =fall[:]->u</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> v =fall[:]->v</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> dbz =fall[:]->dbz</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> rho =fall[:]->rho</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> w =fall[:]->w</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> t =fall[:]->t</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> q =fall[:]->q</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> p =fall[:]->p</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier;min-height:22px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures">;=====================================================================</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ; output variables directly</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier;min-height:22px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> filo = "RI_subset_cat.nc"</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> system("rm -f filo")</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> setfileoption("nc","Format","NetCDF4")</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf = addfile(filo,"c")</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier;min-height:22px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ; make time an UNLIMITED dimension</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> filedimdef(ncdf,"time",-1,True)</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier;min-height:22px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->u = u</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->v = v</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->w = w</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->t = t</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->q = q</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->p = p</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->dbz= dbz</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier"><span style="font-variant-ligatures:no-common-ligatures"> ncdf->rho= rho</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Courier;min-height:22px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div></div><div><br></div><div><br></div><div><br><div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="auto" style="overflow-wrap: break-word;"><div style="color:rgb(0,0,0);font-family:Palatino;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">Dr. Saiprasanth Bhalachandran (Sai)<br>Department of Earth System Science,<br>Stanford University<br>Website: <a href="https://sites.google.com/view/saiprasanth/" target="_blank">https://sites.google.com/view/saiprasanth/</a></div><div style="color:rgb(0,0,0);font-family:Palatino;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><br><br><br></div></div></div>
</div>
<br></div></div>_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a></blockquote></div>