[ncl-talk] Subsetting MACAv2 files

Fri Sep 7 10:54:53 MDT 2018

Hi Toni,

I don't know the answer to your question for certain, but compression might
indeed be the issue. You can test this readily by running something like:

   ncl_convert2nc ..your-netcdf-filename.nc -nc4 -cl 5

and comparing the resultant file to the input. The "-cl 5" specifies a
compression level, a value ranging from 1 to 9 (the docs say that values
above 5 yield only marginally better compression).

If this is the case, you can compress your files at the time you write them
with a "CompressLevel" setfileoption:

    http://ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml

Hope that helps...
Rick

On Fri, Sep 7, 2018 at 9:48 AM, Toni Klemm <toni-klemm at tamu.edu> wrote:

> Good morning,
>
> I created spatial subsets from MACAv2 downscaled climate projections for
> the continental US (CONUS), however, the subsetted files have the same file
> as the originals (ca. 1.5 GB), even though the subset area only covers
> about half of CONUS (I checked). I couldn’t attach the files, but I
> attached the NCL script below. The NCL script I use is adapted from one I
> used to create subsets of vegetation projections for which the file size
> afterwards was about half, as one would expect. Does anyone have ideas for
> why the size isn’t reduced? Is the original MACAv2 data somehow compressed
> and gets uncompressed in the process? Any advice is much appreciated.
>
> Thank you very much!
>
> Best,
> Toni
>
>
>
> ; ---------------------------------------------------
>
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>
> ; ---------------------------------------------------
>
> begin
>
> RCP_in_list = [/"rcp45","rcp85"/]
>
> RCP_out_list = [/"rcp_45","rcp_85"/]
>
> model_list = [/"CCSM4","GFDL-ESM2M","HadGEM2-ES365","IPSL-CM5A-LR",
> "MRI-CGCM3"/]
>
> variable_list = [/"ppt","tdmean","tmax","tmin"/]
>
> do h = 0, ListCount(RCP_in_list) - 1
>
>   do j = 0, ListCount(model_list) - 1
>
>     do k = 0, ListCount(variable_list) - 1
>
>
>   ; ***********************************************
>   ; READ IN MACA FILE
>   ; ***********************************************
>
>   print("1 - Read in " + RCP_in_list[h](0) + "/" + model_list[j](0) + "/"
> + variable_list[k](0) + ".nc")
>
>   MACA_f = addfile("~/NIFA/MACAv2/MACA_" + RCP_in_list[h](0) + "/" +
> model_list[j](0) + "/" + variable_list[k](0) + ".nc", "r")
>
>
>   ; ***********************************************
>   ; READ IN VARIABLES
>   ; ***********************************************
>
>   print("2 - Read in variable")
>
>   if(variable_list[k](0) .eq. "ppt") then
>   variable = MACA_f->ppt(:,:,:)
>   end if
>
>   if(variable_list[k](0) .eq. "tdmean") then
>   variable = MACA_f->tdmean(:,:,:)
>   end if
>
>   if(variable_list[k](0) .eq. "tmax") then
>   variable = MACA_f->tmax(:,:,:)
>   end if
>
>   if(variable_list[k](0) .eq. "tmin") then
>   variable = MACA_f->tmin(:,:,:)
>   end if
>
>   lat = MACA_f->lat
>   lon = MACA_f->lon
>   time = MACA_f->time
>   description = variable at long_name
>
>
>   ; ***********************************************
>   ; CREATE US-SUBSET OF THE GLOBAL DATA
>   ; ***********************************************
>
>   print("3 - Create lat/lon subset")
>
>   ; let's choose latitudes of
>     latind = (/25,50/)
>
>   ; let's choose longitudes of
>     lonind = (/-118,-92/)
>
>   lti = ind_nearest_coord(latind,lat,0)  ; lti, lni = variables,
> ind_nearest_coord = pre-defined functions to determine the indices of
> locations closest to the coordinate array, needed later to define
> var_subdomain_3D
>   lni = ind_nearest_coord(lonind,lon,0)
>
>   lat_subdomain = lat(lti(0):lti(1))
>   lon_subdomain = lon(lni(0):lni(1))
>
>   var_subdomain_3D = variable(:,lti(0):lti(1),lni(0):lni(1))  ; define
> the subdomain based on latind and lonind
>
>
>   ; ***********************************************
>   ; 3 - WRITE THE SUBSET INTO NEW netCDF FILE
>   ; ***********************************************
>
>   print("4 - Write subset into new netCDF file")
>
>     netCDF1 = True    ; Output format is NetCDF
>
>   if (netCDF1) then
>     diro = "~/NIFA/MACAv2/subset_lat_lon/" + RCP_out_list[h](0) + "/" +
> model_list[j](0) + "/"
>     filo = variable_list[k](0) + "_lat_lon.nc"
>
>   end if
>
>   setfileoption("nc","Format","LargeFile")
>
>   if (netCDF1) then
>     system("/bin/rm -f "+ diro + filo)   ; removes old files with the same
> name if they are present
>     ncdf = addfile(diro + filo,"c")
>
>     setfileoption(ncdf,"DefineMode",True)
>
>     ; create attributes
>     fAtt = True
>     fAtt at title = "MACAv2 data, RCP: " + RCP_out_list[h](0) + " Climate
> model: " + model_list[j](0) + " Variable: " + variable_list[k](0)
>     fAtt at description = description
>     fAtt at Conventions = "None"
>     fAtt at creation_date = systemfunc("date")
>
>     fileattdef(ncdf,fAtt)
>
>     dimNames = (/"time","lat","lon"/)
>     dimSizes = (/dimsizes(time),dimsizes(lat_subdomain),dimsizes(lon_
> subdomain)/)
>     dimUnlim = (/True,False,False/)
>
>     filedimdef(ncdf,dimNames,dimSizes,dimUnlim)
>
>     filevardef(ncdf,"time",typeof(time),(/"time"/))
>     filevardef(ncdf,"lat",typeof(lat_subdomain),(/"lat"/))
>     filevardef(ncdf,"lon",typeof(lon_subdomain),(/"lon"/))
>     filevardef(ncdf,"var",typeof(var_subdomain_3D),(/"time","lat","lon"/))
>
>     ; now write all the variables to the file
>
>     ncdf->time = (time)
>     ncdf->lat  = (lat_subdomain)
>     ncdf->lon  = (lon_subdomain)
>     ncdf->var  = (var_subdomain_3D)
>
>   end if
>
>   print("**** DONE ****")
>   print("")
>
>   delete([/variable,var_subdomain_3D,lat,lon/])
>
>
>     end do   ; end variable list loop
>
>   end do  ; end model list loop
>
> end do  ; end RCP list loop
>
> end
>
>
>
>
> *Toni Klemm, Ph.D.*Postdoctoral Research Associate
> Department of Ecosystem Science and Management
> College of Agriculture and Life Sciences
> Texas A&M University, College Station, TX
> Contributor to the Early Career Climate Forum <http://www.eccforum.org>
> www.toni-klemm.de | @toniklemm <http://twitter.com/toniklemm>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180907/454ce7b2/attachment.html>