[ncl-talk] Subsetting MACAv2 files

Fri Sep 7 11:19:15 MDT 2018

Thanks Rick, that worked! Did you mean "-nc4c”? NCL didn’t know what to do with “-nc4”. I tried -nc4c and it compressed it to 923 MB (from 1.6 GB), which sounds about right. 

Thanks again!

Toni

Toni Klemm, Ph.D.
Postdoctoral Research Associate
Department of Ecosystem Science and Management
College of Agriculture and Life Sciences
Texas A&M University, College Station, TX
Contributor to the Early Career Climate Forum <http://www.eccforum.org/>
www.toni-klemm.de <http://www.toni-klemm.de/> | @toniklemm <http://twitter.com/toniklemm>

> On Sep 7, 2018, at 11:54 AM, Rick Brownrigg <brownrig at ucar.edu> wrote:
> 
> Hi Toni,
> 
> I don't know the answer to your question for certain, but compression might indeed be the issue. You can test this readily by running something like:
> 
>    ncl_convert2nc ..your-netcdf-filename.nc <https://urldefense.proofpoint.com/v2/url?u=http-3A__your-2Dnetcdf-2Dfilename.nc&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=LrZa8sKEkQ--fvUv2VN0gZE3e_uXFNLaZiy0wbyMXZk&e=> -nc4 -cl 5 
> 
> and comparing the resultant file to the input. The "-cl 5" specifies a compression level, a value ranging from 1 to 9 (the docs say that values above 5 yield only marginally better compression).
> 
> If this is the case, you can compress your files at the time you write them with a "CompressLevel" setfileoption:
> 
>     http://ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml <https://urldefense.proofpoint.com/v2/url?u=http-3A__ncl.ucar.edu_Document_Functions_Built-2Din_setfileoption.shtml&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=g3LC1n8vazGvUERiEHvorZ2-yxr1iPdmTkZ67YzANeM&e=>
> 
> Hope that helps...
> Rick
> 
> 
> On Fri, Sep 7, 2018 at 9:48 AM, Toni Klemm <toni-klemm at tamu.edu <mailto:toni-klemm at tamu.edu>> wrote:
> Good morning,
> 
> I created spatial subsets from MACAv2 downscaled climate projections for the continental US (CONUS), however, the subsetted files have the same file as the originals (ca. 1.5 GB), even though the subset area only covers about half of CONUS (I checked). I couldn’t attach the files, but I attached the NCL script below. The NCL script I use is adapted from one I used to create subsets of vegetation projections for which the file size afterwards was about half, as one would expect. Does anyone have ideas for why the size isn’t reduced? Is the original MACAv2 data somehow compressed and gets uncompressed in the process? Any advice is much appreciated.
> 
> Thank you very much!
> 
> Best,
> Toni
> 
> 
> 
> ; ---------------------------------------------------
> 
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> 
> ; ---------------------------------------------------
> 
> begin
> 
> RCP_in_list = [/"rcp45","rcp85"/]
> 
> RCP_out_list = [/"rcp_45","rcp_85"/]
> 
> model_list = [/"CCSM4","GFDL-ESM2M","HadGEM2-ES365","IPSL-CM5A-LR","MRI-CGCM3"/]
> 
> variable_list = [/"ppt","tdmean","tmax","tmin"/]
> 
> do h = 0, ListCount(RCP_in_list) - 1 
> 
>   do j = 0, ListCount(model_list) - 1 
> 
>     do k = 0, ListCount(variable_list) - 1 
> 
> 
>   ; ***********************************************
>   ; READ IN MACA FILE
>   ; ***********************************************
> 
>   print("1 - Read in " + RCP_in_list[h](0) + "/" + model_list[j](0) + "/" + variable_list[k](0) + ".nc")
> 
>   MACA_f = addfile("~/NIFA/MACAv2/MACA_" + RCP_in_list[h](0) + "/" + model_list[j](0) + "/" + variable_list[k](0) + ".nc", "r")
> 
> 
>   ; ***********************************************
>   ; READ IN VARIABLES
>   ; ***********************************************
> 
>   print("2 - Read in variable")
> 
>   if(variable_list[k](0) .eq. "ppt") then
>   	variable = MACA_f->ppt(:,:,:)
>   end if
> 
>   if(variable_list[k](0) .eq. "tdmean") then
>   	variable = MACA_f->tdmean(:,:,:)
>   end if
> 
>   if(variable_list[k](0) .eq. "tmax") then
>   	variable = MACA_f->tmax(:,:,:)
>   end if
> 
>   if(variable_list[k](0) .eq. "tmin") then
>   	variable = MACA_f->tmin(:,:,:)
>   end if
> 
>   lat = MACA_f->lat
>   lon = MACA_f->lon
>   time = MACA_f->time
>   description = variable at long_name
>   
> 
>   ; ***********************************************
>   ; CREATE US-SUBSET OF THE GLOBAL DATA
>   ; ***********************************************
> 
>   print("3 - Create lat/lon subset")
> 
>   ; let's choose latitudes of 
>     latind = (/25,50/)
> 
>   ; let's choose longitudes of 
>     lonind = (/-118,-92/)
> 
>   lti = ind_nearest_coord(latind,lat,0)  ; lti, lni = variables, ind_nearest_coord = pre-defined functions to determine the indices of locations closest to the coordinate array, needed later to define var_subdomain_3D
>   lni = ind_nearest_coord(lonind,lon,0)
> 
>   lat_subdomain = lat(lti(0):lti(1))
>   lon_subdomain = lon(lni(0):lni(1))
> 
>   var_subdomain_3D = variable(:,lti(0):lti(1),lni(0):lni(1))  ; define the subdomain based on latind and lonind
> 
> 
>   ; ***********************************************
>   ; 3 - WRITE THE SUBSET INTO NEW netCDF FILE
>   ; ***********************************************
> 
>   print("4 - Write subset into new netCDF file")
> 
>     netCDF1 = True    ; Output format is NetCDF
> 
>   if (netCDF1) then
>     diro = "~/NIFA/MACAv2/subset_lat_lon/" + RCP_out_list[h](0) + "/" + model_list[j](0) + "/"
>     filo = variable_list[k](0) + "_lat_lon.nc <https://urldefense.proofpoint.com/v2/url?u=http-3A__lat-5Flon.nc&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=pyudTkkJ_ZtJZ0A00Cn8Oto_19IBuBcBopEVV3y1Ntg&e=>"
> 
>   end if
> 
>   setfileoption("nc","Format","LargeFile")
> 
>   if (netCDF1) then
>     system("/bin/rm -f "+ diro + filo)   ; removes old files with the same name if they are present
>     ncdf = addfile(diro + filo,"c")
> 
>     setfileoption(ncdf,"DefineMode",True)
> 
>     ; create attributes
>     fAtt = True
>     fAtt at title = "MACAv2 data, RCP: " + RCP_out_list[h](0) + " Climate model: " + model_list[j](0) + " Variable: " + variable_list[k](0)
>     fAtt at description = description
>     fAtt at Conventions = "None"
>     fAtt at creation_date = systemfunc("date")
> 
>     fileattdef(ncdf,fAtt)
> 
>     dimNames = (/"time","lat","lon"/)
>     dimSizes = (/dimsizes(time),dimsizes(lat_subdomain),dimsizes(lon_subdomain)/)
>     dimUnlim = (/True,False,False/)
> 
>     filedimdef(ncdf,dimNames,dimSizes,dimUnlim)
> 
>     filevardef(ncdf,"time",typeof(time),(/"time"/))
>     filevardef(ncdf,"lat",typeof(lat_subdomain),(/"lat"/))
>     filevardef(ncdf,"lon",typeof(lon_subdomain),(/"lon"/))
>     filevardef(ncdf,"var",typeof(var_subdomain_3D),(/"time","lat","lon"/))
> 
>     ; now write all the variables to the file
> 
>     ncdf->time = (time)
>     ncdf->lat  = (lat_subdomain)
>     ncdf->lon  = (lon_subdomain)
>     ncdf->var  = (var_subdomain_3D)
> 
>   end if
> 
>   print("**** DONE ****")
>   print("")
> 
>   delete([/variable,var_subdomain_3D,lat,lon/])
> 
> 
>     end do   ; end variable list loop
> 
>   end do  ; end model list loop
> 
> end do  ; end RCP list loop
> 
> end
> 
> 
> 
> Toni Klemm, Ph.D.
> Postdoctoral Research Associate
> Department of Ecosystem Science and Management
> College of Agriculture and Life Sciences
> Texas A&M University, College Station, TX
> Contributor to the Early Career Climate Forum <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.eccforum.org&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=AixpJUUT-oh2qe-1oZm0KmgnNPgGm6gV2a8OMPbP_nE&e=>
> www.toni-klemm.de <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.toni-2Dklemm.de&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=zYTsFJvCJqSflzvqJE0ScRFtKyzlhGkgZN_hnYwPsL0&e=> | @toniklemm <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_toniklemm&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=-3Gsq-LsjMwDbk7I1HXDznu58ln6FghJ7xJMOY5ZNYk&e=>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ucar.edu_mailman_listinfo_ncl-2Dtalk&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=YOoxLB9BKZ36rV0v0C8l6AEFquu5miLznCSGyTTVc64&e=>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180907/3548149d/attachment.html>