[ncl-talk] Updating NetCDF File frame by frame vs all at once

[Adam Phillips]

Hi Joe,
I think what is happening is that when you use this coding:
  wrfpost = addfile(dir_out+file_out,"c")    ;create new netCDF file
do n = 0, nTime-1
     print(n)
     wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)    <- Errors
out
 end do

the slp array is not preallocated space on the output .nc file, so NCL
doesn't know to write the entire 3D slp array the first time you write to
the slp array. The possible way around this is to create the .nc file the
efficient way in NCL, where arrays are preallocated. See method #2 here:
http://www.ncl.ucar.edu/Applications/write_netcdf.shtml

<http://www.ncl.ucar.edu/Applications/write_netcdf.shtml>
I have not created a .nc file this way, writing partially to an array like
what I am proposing you do, but I do not see any reason why it wouldn't
work. As your script is rather long, I would recommend trying a write and
run a short test script attempting to do this offline, before implementing
it into your script.
Hope that helps!
Adam



On Wed, Apr 21, 2021 at 12:55 PM Joseph Zambon via ncl-talk <
ncl-talk at mailman.ucar.edu> wrote:

> NCL-Talk,
>
> I'm attempting to modify the wrfout_to_cf.ncl
> <https://sundowner.colorado.edu/wrfout_to_cf/release_notes.html> script
> by Matt Higgins to loop over time.  The existing script calculates
> variables like SLP by ingesting the entire dataset and then spitting out
> the result.  For smaller domains / fewer frames per outfile, this works
> fine.  For my outfile, I have 601 timesteps and a domain of 1664x1544x50
> meaning each 4-D variable takes up ~288GB, which is way too much to work
> with efficiently in memory.
>
> I'm trying to re-structure the code so that it loops over time.  E.g. at
> each timestep for SLP it will calculate SLP from 3-D fields (x,y,z) and
> then output a 2-D array (x,y) to save to a NetCDF.  The problem comes when
> I try to save to NetCDF, I get an error...
> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 461 in
> file wrfout_to_cf_timefix.ncl
>
> when code is executed as...
>         do n = 0, nTime-1
>           print(n)
>           ;slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
>           wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)    <-
> Errors out
>         end do
>         ;wrfpost->slp =slp
>
> wrfpost = output file.
> slp = netcdf variable (601,1664,1544)
> n = timestep
>
> However, when I (less efficiently) run over the loop of 601 timesteps and
> save them to a 3-D SLP variable, then dump them into the NetCDF file, it
> works fine...
>         do n = 0, nTime-1
>           print(n)
>           slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
>           ;wrfpost->slp(n,:,:)=(/wrf_user_getvar(wrfout,"slp",n)/)
>         end do
>         wrfpost->slp =slp   <- Works fine
>
> AFAIK, the dimsizes are lined up correctly.  Ideally, I would execute this
> code over each timestep, save the result to the NetCDF file, then rerun so
> at any point I only have 3-D/2-D and not 4-D/3-D arrays to handle to save
> memory.  I know I could ncks the wrfout file into smaller chunks but I'd
> rather figure out what's going on than work around it.
>
> The entire script is attached, for the sake of simplicity ignore the other
> 4-D variables.  Once I have SLP figured out, the others should fall into
> place.
>
> Please let me know what I'm doing that's stupid.
>
> Thanks!
>
> -Joe
>
>
>
>
> Joseph B. Zambon, Ph.D.
> Research Assistant Professor
>
> Ocean Observing and Modeling Group
> Department of Marine, Earth and Atmospheric Sciences
> North Carolina State University
> Jordan Hall rm 4140
> 919.515.7928
> jbzambon at ncsu.edu
> www.joezambon.com
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at mailman.ucar.edu
> List instructions, subscriber options, unsubscribe:
> https://mailman.ucar.edu/mailman/listinfo/ncl-talk



-- 
Adam Phillips
Associate Scientist,  Climate and Global Dynamics Laboratory, NCAR
www.cgd.ucar.edu/staff/asphilli/   303-497-1726

<http://www.cgd.ucar.edu/staff/asphilli>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210422/caa56dbc/attachment.html>