[ncl-talk] Handling large data files

[Prashanth Bhalachandran]

Dear NCL-ers, 
I have a question regarding handling large amounts of data in NCL. 

Currently, I have 41 Netcdf files that are each about 4.5 GBs. Each file contains eight variables that are  
[Time | 10] x [bottom_top | 60] x [south_north | 480] x [west_east_stag | 481]

So far, I have managed to handle each of these files separately, but now I need to append them. I am using the following code but the data size is so large that the program is getting constantly killed even on servers (I tried requesting around 80 GB of memory). Is there a more efficient way to concatenate these files around the time dimension? I am attaching the code that I currently use below. To make the matters worse, I realized that some of my files are corrupt and need to be regenerated. However, I am unable to weed out which ones are corrupt without the below program running fully (that is currently difficult due to memory issues). 

Can anyone help me out here, please? 

Sai 


all_files = systemfunc("ls /scratch/groups/oneillm/Nature_Runs/HNR1/domain4/RI_subsets/subset_*.nc")
        setfileoption("nc","Format","NetCDF4")
        fall=addfiles(all_files, "r")
        ListSetType(fall, "cat")


        u =fall[:]->u
        v =fall[:]->v
        dbz =fall[:]->dbz
        rho =fall[:]->rho
        w =fall[:]->w
        t =fall[:]->t
        q =fall[:]->q
        p =fall[:]->p

;=====================================================================
      ; output variables directly

         filo = "RI_subset_cat.nc"
         system("rm -f filo")
         setfileoption("nc","Format","NetCDF4")
         ncdf = addfile(filo,"c")

      ; make time an UNLIMITED dimension
        filedimdef(ncdf,"time",-1,True)

         ncdf->u  = u
         ncdf->v  = v
         ncdf->w  = w
         ncdf->t  = t
         ncdf->q  = q
         ncdf->p  = p
         ncdf->dbz= dbz
         ncdf->rho= rho




Dr. Saiprasanth Bhalachandran (Sai)
Department of Earth System Science,
Stanford University
Website: https://sites.google.com/view/saiprasanth/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190827/30d63c0c/attachment.html>