<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Dear NCL-ers, <div class="">I have a question regarding handling large amounts of data in NCL. </div><div class=""><br class=""></div><div class="">Currently, I have 41 Netcdf files that are each about 4.5 GBs. Each file contains eight variables that are <span style="color: rgb(180, 36, 25); font-family: Courier;" class=""> </span></div><div class=""><span style="color: rgb(180, 36, 25); font-family: Courier;" class="">[Time | 10] x [bottom_top | 60] x [south_north | 480] x [west_east_stag | 481]</span></div><div class=""><span style="color: rgb(180, 36, 25); font-family: Courier;" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(180, 36, 25);" class="">So far, I have managed to handle each of these files separately, but now I need to append them. I am using the following code but the data size is so large that the program is getting constantly killed even on servers (I tried requesting around 80 GB of memory). Is there a more efficient way to concatenate these files around the time dimension? I am attaching the code that I currently use below. To make the matters worse, I realized that some of my files are corrupt and need to be regenerated. However, I am unable to weed out which ones are corrupt without the below program running fully (that is currently difficult due to memory issues). </span></div><div class=""><span style="caret-color: rgb(180, 36, 25);" class=""><br class=""></span></div><div class=""><span style="caret-color: rgb(180, 36, 25);" class="">Can anyone help me out here, please? </span></div><div class=""><font face="Courier" class=""><span style="caret-color: rgb(180, 36, 25);" class=""><br class=""></span></font></div><div class=""><font face="Courier" class=""><span style="caret-color: rgb(180, 36, 25);" class="">Sai </span></font></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">all_files = systemfunc("ls /scratch/groups/oneillm/Nature_Runs/HNR1/domain4/RI_subsets/subset_*.nc")</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> setfileoption("nc","Format","NetCDF4")</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> fall=addfiles(all_files, "r")</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ListSetType(fall, "cat")</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; min-height: 22px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; min-height: 22px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> u =fall[:]->u</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> v =fall[:]->v</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> dbz =fall[:]->dbz</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rho =fall[:]->rho</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> w =fall[:]->w</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> t =fall[:]->t</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> q =fall[:]->q</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> p =fall[:]->p</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; min-height: 22px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">;=====================================================================</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ; output variables directly</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; min-height: 22px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> filo = "RI_subset_cat.nc"</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> system("rm -f filo")</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> setfileoption("nc","Format","NetCDF4")</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf = addfile(filo,"c")</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; min-height: 22px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ; make time an UNLIMITED dimension</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> filedimdef(ncdf,"time",-1,True)</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; min-height: 22px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->u = u</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->v = v</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->w = w</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->t = t</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->q = q</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->p = p</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->dbz= dbz</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ncdf->rho= rho</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; min-height: 22px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Palatino; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Dr. Saiprasanth Bhalachandran (Sai)<br class="">Department of Earth System Science,<br class="">Stanford University<br class="">Website: <a href="https://sites.google.com/view/saiprasanth/" class="">https://sites.google.com/view/saiprasanth/</a></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Palatino; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class=""><br class=""><br class=""></div></div></div>
</div>
<br class=""></div></body></html>