[ncl-talk] Updating NetCDF File frame by frame vs all at once

[Joseph Zambon]

NCL-Talk,

I'm attempting to modify the wrfout_to_cf.ncl <https://sundowner.colorado.edu/wrfout_to_cf/release_notes.html> script by Matt Higgins to loop over time.  The existing script calculates variables like SLP by ingesting the entire dataset and then spitting out the result.  For smaller domains / fewer frames per outfile, this works fine.  For my outfile, I have 601 timesteps and a domain of 1664x1544x50 meaning each 4-D variable takes up ~288GB, which is way too much to work with efficiently in memory.

I'm trying to re-structure the code so that it loops over time.  E.g. at each timestep for SLP it will calculate SLP from 3-D fields (x,y,z) and then output a 2-D array (x,y) to save to a NetCDF.  The problem comes when I try to save to NetCDF, I get an error...
fatal:["Execute.c":8637]:Execute: Error occurred at or near line 461 in file wrfout_to_cf_timefix.ncl

when code is executed as...
        do n = 0, nTime-1
          print(n)
          ;slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
          wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)    <- Errors out
        end do
        ;wrfpost->slp =slp

wrfpost = output file.
slp = netcdf variable (601,1664,1544)
n = timestep

However, when I (less efficiently) run over the loop of 601 timesteps and save them to a 3-D SLP variable, then dump them into the NetCDF file, it works fine...
        do n = 0, nTime-1
          print(n)
          slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
          ;wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
        end do
        wrfpost->slp =slp   <- Works fine

AFAIK, the dimsizes are lined up correctly.  Ideally, I would execute this code over each timestep, save the result to the NetCDF file, then rerun so at any point I only have 3-D/2-D and not 4-D/3-D arrays to handle to save memory.  I know I could ncks the wrfout file into smaller chunks but I'd rather figure out what's going on than work around it.

The entire script is attached, for the sake of simplicity ignore the other 4-D variables.  Once I have SLP figured out, the others should fall into place.

Please let me know what I'm doing that's stupid.

Thanks!

-Joe




Joseph B. Zambon, Ph.D.
Research Assistant Professor

Ocean Observing and Modeling Group
Department of Marine, Earth and Atmospheric Sciences
North Carolina State University
Jordan Hall rm 4140
919.515.7928
jbzambon at ncsu.edu <mailto:jbzambon at ncsu.edu>
www.joezambon.com <http://www.joezambon.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210421/bea20f25/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wrfout_to_cf_timefix.ncl
Type: application/octet-stream
Size: 21924 bytes
Desc: not available
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210421/bea20f25/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210421/bea20f25/attachment-0001.html>