<div dir="ltr"><div dir="ltr"><div>Hi Toni,<br></div><div><br></div><div>I don't know the answer to your question for certain, but compression might indeed be the issue. You can test this readily by running something like:</div><div><br></div><div> ncl_convert2nc ..<a href="http://your-netcdf-filename.nc">your-netcdf-filename.nc</a> -nc4 -cl 5 <br></div><div><br></div><div>and comparing the resultant file to the input. The "-cl 5" specifies a compression level, a value ranging from 1 to 9 (the docs say that values above 5 yield only marginally better compression).</div><div><br></div><div>If this is the case, you can compress your files at the time you write them with a "CompressLevel" setfileoption:</div><div><br></div><div> <a href="http://ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml">http://ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml</a><br></div><div><br></div><div>Hope that helps...</div><div>Rick<br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 7, 2018 at 9:48 AM, Toni Klemm <span dir="ltr"><<a href="mailto:toni-klemm@tamu.edu" target="_blank">toni-klemm@tamu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Good morning,<div><br></div><div>I created spatial subsets from MACAv2 downscaled climate projections for the continental US (CONUS), however, the subsetted files have the same file as the originals (ca. 1.5 GB), even though the subset area only covers about half of CONUS (I checked). I couldn’t attach the files, but I attached the NCL script below. The NCL script I use is adapted from one I used to create subsets of vegetation projections for which the file size afterwards was about half, as one would expect. Does anyone have ideas for why the size isn’t reduced? Is the original MACAv2 data somehow compressed and gets uncompressed in the process? Any advice is much appreciated.</div><div><br></div><div>Thank you very much!</div><div><br></div><div>Best,</div><div>Toni</div><div><br></div><div><br></div><div><div><br></div><div><span style="font-family:Courier">; ------------------------------<wbr>---------------------</span></div><div><font face="Courier"><br></font></div><div><font face="Courier">load "$NCARG_ROOT/lib/ncarg/<wbr>nclscripts/csm/gsn_code.ncl"</font></div><div><font face="Courier">load "$NCARG_ROOT/lib/ncarg/<wbr>nclscripts/csm/gsn_csm.ncl"</font></div><div><font face="Courier">load "$NCARG_ROOT/lib/ncarg/<wbr>nclscripts/csm/contributed.<wbr>ncl"</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">; ------------------------------<wbr>---------------------</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">begin</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">RCP_in_list = [/"rcp45","rcp85"/]</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">RCP_out_list = [/"rcp_45","rcp_85"/]</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">model_list = [/"CCSM4","GFDL-ESM2M","<wbr>HadGEM2-ES365","IPSL-CM5A-LR",<wbr>"MRI-CGCM3"/]</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">variable_list = [/"ppt","tdmean","tmax","tmin"<wbr>/]</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">do h = 0, ListCount(RCP_in_list) - 1 </font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> do j = 0, ListCount(model_list) - 1 </font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> do k = 0, ListCount(variable_list) - 1 </font></div><div><font face="Courier"><br></font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"> ; READ IN MACA FILE</font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> print("1 - Read in " + RCP_in_list[h](0) + "/" + model_list[j](0) + "/" + variable_list[k](0) + ".nc")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> MACA_f = addfile("~/NIFA/MACAv2/MACA_" + RCP_in_list[h](0) + "/" + model_list[j](0) + "/" + variable_list[k](0) + ".nc", "r")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"> ; READ IN VARIABLES</font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> print("2 - Read in variable")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> if(variable_list[k](0) .eq. "ppt") then</font></div><div><font face="Courier"> <span class="m_7816898501269950989Apple-tab-span" style="white-space:pre-wrap"> </span>variable = MACA_f->ppt(:,:,:)</font></div><div><font face="Courier"> end if</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> if(variable_list[k](0) .eq. "tdmean") then</font></div><div><font face="Courier"> <span class="m_7816898501269950989Apple-tab-span" style="white-space:pre-wrap"> </span>variable = MACA_f->tdmean(:,:,:)</font></div><div><font face="Courier"> end if</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> if(variable_list[k](0) .eq. "tmax") then</font></div><div><font face="Courier"> <span class="m_7816898501269950989Apple-tab-span" style="white-space:pre-wrap"> </span>variable = MACA_f->tmax(:,:,:)</font></div><div><font face="Courier"> end if</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> if(variable_list[k](0) .eq. "tmin") then</font></div><div><font face="Courier"> <span class="m_7816898501269950989Apple-tab-span" style="white-space:pre-wrap"> </span>variable = MACA_f->tmin(:,:,:)</font></div><div><font face="Courier"> end if</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> lat = MACA_f->lat</font></div><div><font face="Courier"> lon = MACA_f->lon</font></div><div><font face="Courier"> time = MACA_f->time</font></div><div><font face="Courier"> description = variable@long_name</font></div><div><font face="Courier"> </font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"> ; CREATE US-SUBSET OF THE GLOBAL DATA</font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> print("3 - Create lat/lon subset")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; let's choose latitudes of </font></div><div><font face="Courier"> latind = (/25,50/)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; let's choose longitudes of </font></div><div><font face="Courier"> lonind = (/-118,-92/)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> lti = ind_nearest_coord(latind,lat,<wbr>0) ; lti, lni = variables, ind_nearest_coord = pre-defined functions to determine the indices of locations closest to the coordinate array, needed later to define var_subdomain_3D</font></div><div><font face="Courier"> lni = ind_nearest_coord(lonind,lon,<wbr>0)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> lat_subdomain = lat(lti(0):lti(1))</font></div><div><font face="Courier"> lon_subdomain = lon(lni(0):lni(1))</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> var_subdomain_3D = variable(:,lti(0):lti(1),lni(<wbr>0):lni(1)) ; define the subdomain based on latind and lonind</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"> ; 3 - WRITE THE SUBSET INTO NEW netCDF FILE</font></div><div><font face="Courier"> ; ******************************<wbr>*****************</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> print("4 - Write subset into new netCDF file")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> netCDF1 = True ; Output format is NetCDF</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> if (netCDF1) then</font></div><div><font face="Courier"> diro = "~/NIFA/MACAv2/subset_lat_lon/<wbr>" + RCP_out_list[h](0) + "/" + model_list[j](0) + "/"</font></div><div><font face="Courier"> filo = variable_list[k](0) + "_<a href="http://lat_lon.nc" target="_blank">lat_lon.nc</a>"</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> end if</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> setfileoption("nc","Format","<wbr>LargeFile")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> if (netCDF1) then</font></div><div><font face="Courier"> system("/bin/rm -f "+ diro + filo) ; removes old files with the same name if they are present</font></div><div><font face="Courier"> ncdf = addfile(diro + filo,"c")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> setfileoption(ncdf,"<wbr>DefineMode",True)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; create attributes</font></div><div><font face="Courier"> fAtt = True</font></div><div><font face="Courier"> fAtt@title = "MACAv2 data, RCP: " + RCP_out_list[h](0) + " Climate model: " + model_list[j](0) + " Variable: " + variable_list[k](0)</font></div><div><font face="Courier"> fAtt@description = description</font></div><div><font face="Courier"> fAtt@Conventions = "None"</font></div><div><font face="Courier"> fAtt@creation_date = systemfunc("date")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> fileattdef(ncdf,fAtt)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> dimNames = (/"time","lat","lon"/)</font></div><div><font face="Courier"> dimSizes = (/dimsizes(time),dimsizes(lat_<wbr>subdomain),dimsizes(lon_<wbr>subdomain)/)</font></div><div><font face="Courier"> dimUnlim = (/True,False,False/)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> filedimdef(ncdf,dimNames,<wbr>dimSizes,dimUnlim)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> filevardef(ncdf,"time",typeof(<wbr>time),(/"time"/))</font></div><div><font face="Courier"> filevardef(ncdf,"lat",typeof(<wbr>lat_subdomain),(/"lat"/))</font></div><div><font face="Courier"> filevardef(ncdf,"lon",typeof(<wbr>lon_subdomain),(/"lon"/))</font></div><div><font face="Courier"> filevardef(ncdf,"var",typeof(<wbr>var_subdomain_3D),(/"time","<wbr>lat","lon"/))</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ; now write all the variables to the file</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> ncdf->time = (time)</font></div><div><font face="Courier"> ncdf->lat = (lat_subdomain)</font></div><div><font face="Courier"> ncdf->lon = (lon_subdomain)</font></div><div><font face="Courier"> ncdf->var = (var_subdomain_3D)</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> end if</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> print("**** DONE ****")</font></div><div><font face="Courier"> print("")</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> delete([/variable,var_<wbr>subdomain_3D,lat,lon/])</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> end do ; end variable list loop</font></div><div><font face="Courier"><br></font></div><div><font face="Courier"> end do ; end model list loop</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">end do ; end RCP list loop</font></div><div><font face="Courier"><br></font></div><div><font face="Courier">end</font></div></div><div><br></div><div><br></div><div><div><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;word-wrap:break-word;line-break:after-white-space"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;word-wrap:break-word;line-break:after-white-space"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;word-wrap:break-word;line-break:after-white-space"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;word-wrap:break-word;line-break:after-white-space"><div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word;line-break:after-white-space"><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><font face="Calibri"><br></font></div><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><font face="Calibri"><b>Toni Klemm, Ph.D.<br></b>Postdoctoral Research Associate</font></div><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Department of Ecosystem Science and Management</div><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">College of Agriculture and Life Sciences</div><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Texas A&M University, College Station, TX</div><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><font face="Calibri">Contributor to the <a href="http://www.eccforum.org" target="_blank">Early Career Climate Forum</a><br><a href="http://www.toni-klemm.de" target="_blank">www.toni-klemm.de</a> | <a href="http://twitter.com/toniklemm" target="_blank">@toniklemm</a></font></div><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><font face="Calibri"><br></font></div><div style="color:rgb(0,0,0);font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><font face="Calibri"><br></font></div></div></div><br class="m_7816898501269950989Apple-interchange-newline"></div><br class="m_7816898501269950989Apple-interchange-newline"></div><br class="m_7816898501269950989Apple-interchange-newline"></div><br class="m_7816898501269950989Apple-interchange-newline"><br class="m_7816898501269950989Apple-interchange-newline">
</div>
<br></div></div><br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>