[ncl-talk] Subsetting MACAv2 files
Alan Brammer
abrammer at albany.edu
Fri Sep 7 16:54:54 MDT 2018
The "-nc4" flag was added to ncl_convert2nc in 6.5.0 but it seems to have
slipped from the user facing docs. It's in the top of the script file
though:
# [-nc4] output a NetCDF 4
# [-nc4c] output a NetCDF 4 "classic" format file
I requested it to be added as the classic version of netcdf4 is sometimes
relatively bad for compression. Therefore you may see further improvement
in file size if you can upgrade to 6.5.0 and utilize that.
or, as mentioned in the recent thread with Douglas Lowe, you could do this
in the script without upgrading to 6.5.0:
by probably replacing :
setfileoption("nc", "Format", "LargeFile")
with
setfileoption("nc", "format", "netcdf4")
setfileoption("nc", "CompressionLevel", 4)
p.s. I'd also vote for not compressing beyond level 1, in my limited
experience it only provides marginal size gains at a significant read/write
expense.
Alan
On Fri, Sep 7, 2018 at 1:19 PM Toni Klemm <toni-klemm at tamu.edu> wrote:
> Thanks Rick, that worked! Did you mean "-nc4c”? NCL didn’t know what to do
> with “-nc4”. I tried -nc4c and it compressed it to 923 MB (from 1.6 GB),
> which sounds about right.
>
> Thanks again!
>
> Toni
>
>
>
>
> *Toni Klemm, Ph.D.*Postdoctoral Research Associate
> Department of Ecosystem Science and Management
> College of Agriculture and Life Sciences
> Texas A&M University, College Station, TX
> Contributor to the Early Career Climate Forum <http://www.eccforum.org>
> www.toni-klemm.de | @toniklemm <http://twitter.com/toniklemm>
>
>
>
>
>
>
>
>
> On Sep 7, 2018, at 11:54 AM, Rick Brownrigg <brownrig at ucar.edu> wrote:
>
> Hi Toni,
>
> I don't know the answer to your question for certain, but compression
> might indeed be the issue. You can test this readily by running something
> like:
>
> ncl_convert2nc ..your-netcdf-filename.nc
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__your-2Dnetcdf-2Dfilename.nc&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=LrZa8sKEkQ--fvUv2VN0gZE3e_uXFNLaZiy0wbyMXZk&e=>
> -nc4 -cl 5
>
> and comparing the resultant file to the input. The "-cl 5" specifies a
> compression level, a value ranging from 1 to 9 (the docs say that values
> above 5 yield only marginally better compression).
>
> If this is the case, you can compress your files at the time you write
> them with a "CompressLevel" setfileoption:
>
> http://ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ncl.ucar.edu_Document_Functions_Built-2Din_setfileoption.shtml&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=g3LC1n8vazGvUERiEHvorZ2-yxr1iPdmTkZ67YzANeM&e=>
>
> Hope that helps...
> Rick
>
>
> On Fri, Sep 7, 2018 at 9:48 AM, Toni Klemm <toni-klemm at tamu.edu> wrote:
>
>> Good morning,
>>
>> I created spatial subsets from MACAv2 downscaled climate projections for
>> the continental US (CONUS), however, the subsetted files have the same file
>> as the originals (ca. 1.5 GB), even though the subset area only covers
>> about half of CONUS (I checked). I couldn’t attach the files, but I
>> attached the NCL script below. The NCL script I use is adapted from one I
>> used to create subsets of vegetation projections for which the file size
>> afterwards was about half, as one would expect. Does anyone have ideas for
>> why the size isn’t reduced? Is the original MACAv2 data somehow compressed
>> and gets uncompressed in the process? Any advice is much appreciated.
>>
>> Thank you very much!
>>
>> Best,
>> Toni
>>
>>
>>
>> ; ---------------------------------------------------
>>
>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>
>> ; ---------------------------------------------------
>>
>> begin
>>
>> RCP_in_list = [/"rcp45","rcp85"/]
>>
>> RCP_out_list = [/"rcp_45","rcp_85"/]
>>
>> model_list =
>> [/"CCSM4","GFDL-ESM2M","HadGEM2-ES365","IPSL-CM5A-LR","MRI-CGCM3"/]
>>
>> variable_list = [/"ppt","tdmean","tmax","tmin"/]
>>
>> do h = 0, ListCount(RCP_in_list) - 1
>>
>> do j = 0, ListCount(model_list) - 1
>>
>> do k = 0, ListCount(variable_list) - 1
>>
>>
>> ; ***********************************************
>> ; READ IN MACA FILE
>> ; ***********************************************
>>
>> print("1 - Read in " + RCP_in_list[h](0) + "/" + model_list[j](0) + "/"
>> + variable_list[k](0) + ".nc")
>>
>> MACA_f = addfile("~/NIFA/MACAv2/MACA_" + RCP_in_list[h](0) + "/" +
>> model_list[j](0) + "/" + variable_list[k](0) + ".nc", "r")
>>
>>
>> ; ***********************************************
>> ; READ IN VARIABLES
>> ; ***********************************************
>>
>> print("2 - Read in variable")
>>
>> if(variable_list[k](0) .eq. "ppt") then
>> variable = MACA_f->ppt(:,:,:)
>> end if
>>
>> if(variable_list[k](0) .eq. "tdmean") then
>> variable = MACA_f->tdmean(:,:,:)
>> end if
>>
>> if(variable_list[k](0) .eq. "tmax") then
>> variable = MACA_f->tmax(:,:,:)
>> end if
>>
>> if(variable_list[k](0) .eq. "tmin") then
>> variable = MACA_f->tmin(:,:,:)
>> end if
>>
>> lat = MACA_f->lat
>> lon = MACA_f->lon
>> time = MACA_f->time
>> description = variable at long_name
>>
>>
>> ; ***********************************************
>> ; CREATE US-SUBSET OF THE GLOBAL DATA
>> ; ***********************************************
>>
>> print("3 - Create lat/lon subset")
>>
>> ; let's choose latitudes of
>> latind = (/25,50/)
>>
>> ; let's choose longitudes of
>> lonind = (/-118,-92/)
>>
>> lti = ind_nearest_coord(latind,lat,0) ; lti, lni = variables,
>> ind_nearest_coord = pre-defined functions to determine the indices of
>> locations closest to the coordinate array, needed later to define
>> var_subdomain_3D
>> lni = ind_nearest_coord(lonind,lon,0)
>>
>> lat_subdomain = lat(lti(0):lti(1))
>> lon_subdomain = lon(lni(0):lni(1))
>>
>> var_subdomain_3D = variable(:,lti(0):lti(1),lni(0):lni(1)) ; define
>> the subdomain based on latind and lonind
>>
>>
>> ; ***********************************************
>> ; 3 - WRITE THE SUBSET INTO NEW netCDF FILE
>> ; ***********************************************
>>
>> print("4 - Write subset into new netCDF file")
>>
>> netCDF1 = True ; Output format is NetCDF
>>
>> if (netCDF1) then
>> diro = "~/NIFA/MACAv2/subset_lat_lon/" + RCP_out_list[h](0) + "/" +
>> model_list[j](0) + "/"
>> filo = variable_list[k](0) + "_lat_lon.nc
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lat-5Flon.nc&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=pyudTkkJ_ZtJZ0A00Cn8Oto_19IBuBcBopEVV3y1Ntg&e=>
>> "
>>
>> end if
>>
>> setfileoption("nc","Format","LargeFile")
>>
>> if (netCDF1) then
>> system("/bin/rm -f "+ diro + filo) ; removes old files with the
>> same name if they are present
>> ncdf = addfile(diro + filo,"c")
>>
>> setfileoption(ncdf,"DefineMode",True)
>>
>> ; create attributes
>> fAtt = True
>> fAtt at title = "MACAv2 data, RCP: " + RCP_out_list[h](0) + " Climate
>> model: " + model_list[j](0) + " Variable: " + variable_list[k](0)
>> fAtt at description = description
>> fAtt at Conventions = "None"
>> fAtt at creation_date = systemfunc("date")
>>
>> fileattdef(ncdf,fAtt)
>>
>> dimNames = (/"time","lat","lon"/)
>> dimSizes =
>> (/dimsizes(time),dimsizes(lat_subdomain),dimsizes(lon_subdomain)/)
>> dimUnlim = (/True,False,False/)
>>
>> filedimdef(ncdf,dimNames,dimSizes,dimUnlim)
>>
>> filevardef(ncdf,"time",typeof(time),(/"time"/))
>> filevardef(ncdf,"lat",typeof(lat_subdomain),(/"lat"/))
>> filevardef(ncdf,"lon",typeof(lon_subdomain),(/"lon"/))
>> filevardef(ncdf,"var",typeof(var_subdomain_3D),(/"time","lat","lon"/))
>>
>> ; now write all the variables to the file
>>
>> ncdf->time = (time)
>> ncdf->lat = (lat_subdomain)
>> ncdf->lon = (lon_subdomain)
>> ncdf->var = (var_subdomain_3D)
>>
>> end if
>>
>> print("**** DONE ****")
>> print("")
>>
>> delete([/variable,var_subdomain_3D,lat,lon/])
>>
>>
>> end do ; end variable list loop
>>
>> end do ; end model list loop
>>
>> end do ; end RCP list loop
>>
>> end
>>
>>
>>
>>
>> *Toni Klemm, Ph.D.*Postdoctoral Research Associate
>> Department of Ecosystem Science and Management
>> College of Agriculture and Life Sciences
>> Texas A&M University, College Station, TX
>> Contributor to the Early Career Climate Forum
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.eccforum.org&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=AixpJUUT-oh2qe-1oZm0KmgnNPgGm6gV2a8OMPbP_nE&e=>
>> www.toni-klemm.de
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.toni-2Dklemm.de&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=zYTsFJvCJqSflzvqJE0ScRFtKyzlhGkgZN_hnYwPsL0&e=>
>> | @toniklemm
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_toniklemm&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=-3Gsq-LsjMwDbk7I1HXDznu58ln6FghJ7xJMOY5ZNYk&e=>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ucar.edu_mailman_listinfo_ncl-2Dtalk&d=DwMFaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=WsLJ5ywbW3XnTzqo_ETV62H6s1a5bcGgnrpl7xPRfic&m=p6-SSBOdy_594JgnYjtJAv84FspJdaPUuzpZM1rtses&s=YOoxLB9BKZ36rV0v0C8l6AEFquu5miLznCSGyTTVc64&e=>
>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180907/7bf69745/attachment.html>
More information about the ncl-talk
mailing list