[ncl-talk] Updating NetCDF File frame by frame vs all at once

[Will Hobbs]

Hi Joe

I’m late to this party but I have done this kind of thing in the past a lot, and Adam’s correct – you need to predefine the variable and its dimensions BEFORE you write the actual data, so it’ll look something like this (adjust actual names as necessary):

wrfpost = addfile(dir_out+file_out,"c")    ;create new netCDF file

dimNam = (/”time”,”lat”,”lon”/)
dimSiz = (/-1,nlat,nlon/)

filedimdef(wrfpost, dimNam, dimSiz,(/True,False,False/))


dList = [/time, lat, lon/]  ;I’,m lazy so I loop through all the dimension defs
do I = 0, 2
filevardef(wrfpost, dimNam(i),typeof(dList[i]),dimNam(i))
filevarattdef(wrfpost, dimNam(i), dList[i])
end do

slpAtt = wrf_user_getvar(wrfout,"slp",0)/)    ;get attributes, vartype from one of the files
filevardef(wrfpost, “slp”,typeof(slpAtt), dimNams)
filevarattdef(wrfpost, “slp”, slpAtt)

wrfpost->slp(0,:,:)=(/slpAtt/)  ;might as well use it since you read it!

do n = 1, nTime-1
     wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
 end do

Will

From: ncl-talk <ncl-talk-bounces at mailman.ucar.edu> on behalf of Adam Phillips via ncl-talk <ncl-talk at mailman.ucar.edu>
Reply to: Adam Phillips <asphilli at ucar.edu>
Date: Friday, 23 April 2021 at 2:24 am
To: Joseph Zambon <jbzambon at ncsu.edu>
Cc: ncl-talk <ncl-talk at mailman.ucar.edu>
Subject: Re: [ncl-talk] Updating NetCDF File frame by frame vs all at once

Hi Joe,
I think what is happening is that when you use this coding:
  wrfpost = addfile(dir_out+file_out,"c")    ;create new netCDF file
do n = 0, nTime-1
     print(n)
     wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)    <- Errors out
 end do

the slp array is not preallocated space on the output .nc file, so NCL doesn't know to write the entire 3D slp array the first time you write to the slp array. The possible way around this is to create the .nc file the efficient way in NCL, where arrays are preallocated. See method #2 here:
http://www.ncl.ucar.edu/Applications/write_netcdf.shtml
<http://www.ncl.ucar.edu/Applications/write_netcdf.shtml>

<http://www.ncl.ucar.edu/Applications/write_netcdf.shtml>
I have not created a .nc file this way, writing partially to an array like what I am proposing you do, but I do not see any reason why it wouldn't work. As your script is rather long, I would recommend trying a write and run a short test script attempting to do this offline, before implementing it into your script.
Hope that helps!
Adam



On Wed, Apr 21, 2021 at 12:55 PM Joseph Zambon via ncl-talk <ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>> wrote:
NCL-Talk,

I'm attempting to modify the wrfout_to_cf.ncl<https://sundowner.colorado.edu/wrfout_to_cf/release_notes.html> script by Matt Higgins to loop over time.  The existing script calculates variables like SLP by ingesting the entire dataset and then spitting out the result.  For smaller domains / fewer frames per outfile, this works fine.  For my outfile, I have 601 timesteps and a domain of 1664x1544x50 meaning each 4-D variable takes up ~288GB, which is way too much to work with efficiently in memory.

I'm trying to re-structure the code so that it loops over time.  E.g. at each timestep for SLP it will calculate SLP from 3-D fields (x,y,z) and then output a 2-D array (x,y) to save to a NetCDF.  The problem comes when I try to save to NetCDF, I get an error...
fatal:["Execute.c":8637]:Execute: Error occurred at or near line 461 in file wrfout_to_cf_timefix.ncl

when code is executed as...
        do n = 0, nTime-1
          print(n)
          ;slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
          wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)    <- Errors out
        end do
        ;wrfpost->slp =slp

wrfpost = output file.
slp = netcdf variable (601,1664,1544)
n = timestep

However, when I (less efficiently) run over the loop of 601 timesteps and save them to a 3-D SLP variable, then dump them into the NetCDF file, it works fine...
        do n = 0, nTime-1
          print(n)
          slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
          ;wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
        end do
        wrfpost->slp =slp   <- Works fine
AFAIK, the dimsizes are lined up correctly.  Ideally, I would execute this code over each timestep, save the result to the NetCDF file, then rerun so at any point I only have 3-D/2-D and not 4-D/3-D arrays to handle to save memory.  I know I could ncks the wrfout file into smaller chunks but I'd rather figure out what's going on than work around it.

The entire script is attached, for the sake of simplicity ignore the other 4-D variables.  Once I have SLP figured out, the others should fall into place.

Please let me know what I'm doing that's stupid.

Thanks!

-Joe



Joseph B. Zambon, Ph.D.
Research Assistant Professor

Ocean Observing and Modeling Group
Department of Marine, Earth and Atmospheric Sciences
North Carolina State University
Jordan Hall rm 4140
919.515.7928
jbzambon at ncsu.edu<mailto:jbzambon at ncsu.edu>
www.joezambon.com<http://www.joezambon.com>

_______________________________________________
ncl-talk mailing list
ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>
List instructions, subscriber options, unsubscribe:
https://mailman.ucar.edu/mailman/listinfo/ncl-talk<https://mailman.ucar.edu/mailman/listinfo/ncl-talk>


--
Adam Phillips
Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR
www.cgd.ucar.edu/staff/asphilli/<http://www.cgd.ucar.edu/staff/asphilli>   303-497-1726



This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210422/e00e0a5e/attachment.html>