<div dir="ltr"><div class="gmail_default" style="font-size:small">Hi all,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">I'm responding to this old ESMF question at the request of somebody who found this thread. but not a satisfactory answer.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">If you are trying to regrid really large variables using the NCL-ESMF regridding suite, then you may need to set one of the the "NetCDFType" options to "netcdf4" which allows you to write > than 4 GB to a file.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">For example, if your source grid is really large, and ESMF_regrid gives you an error:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px">ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints</span><br></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px"><br></span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px">then try setting:</span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px"><br></span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px">Opt@SrcNetCDFType = "netcdf4"</span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px"><br></span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px">There are similar options for the destination file "DstNetCDFType" and weights file "WgtNetCDFType".</span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px"><br></span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px">Noel, my hope is that you found this out long before now!</span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px"><br></span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px">--Mary</span></div><div class="gmail_default" style="font-size:small"><span style="font-size:12.8px"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 19, 2013 at 8:05 AM, Mary Haley <span dir="ltr"><<a href="mailto:haley@ucar.edu" target="_blank">haley@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div>Hi Noel,<div><br></div><div>I see what the problem is, and I'm not sure yet how to fix it.</div><span class=""><div><br></div><div>The issue is in order to do ESMF regridding, we need to write an intermediate NetCDF SCRIP file. This file will have two double variables of size grid_size, which is equal to nlat * nlon.</div><div><br></div></span><div>In your case:</div><div><br></div><div> nlat/nlon = 16,800/43,200</div><div> grid_size = nlat*lon = 725760000</div><div> variable to be written to the file = 725760000*8 = 5,806,080,000 bytes</div><div><br></div><div>This is > 4 GB. This violates a NetCDF constraint that individual variables be less than 4 GB.</div><span class=""><div><br></div><div>My first thought is that if you can create a NetCDF-4 file, then you won't be limited by this constraint. The problem is that the ESMF regridding tool we use appears to have a constraint that the intermediate NetCDF files be NetCDF-3 (classic or 64-bit offset) files, and not NetCDF-4.</div><div><br></div><div>I will check into this and get back to you.</div><div><br></div><div>--Mary</div><div><br></div></span><div><br><div><div><div class="h5"><div>On Mar 7, 2013, at 10:27 AM, Noel Aloysius wrote:</div><br class="m_-8598619938295430762Apple-interchange-newline"></div></div><blockquote type="cite"><div><div class="h5">Hi Mary,<br><br>The data file is uploaded to the ftp server - <a href="ftp://ftp.cgd.ucar.edu/incoming/MOD17A3_Science_GPP_2001.zip" target="_blank">ftp://ftp.cgd.ucar.edu/<wbr>incoming/MOD17A3_Science_GPP_<wbr>2001.zip</a>.<br><br>The script is attached.<br>
<br>I want to regrid ftom 1km to 0.5x0.5deg lat/lon.<br><br>Thanks,<br>Noel<br><br><br clear="all"><div>Noel Aloysius</div>
<br><br><div class="gmail_quote">On Thu, Mar 7, 2013 at 9:56 AM, Mary Haley <span dir="ltr"><<a href="mailto:haley@ucar.edu" target="_blank">haley@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Noel,<br>
<br>
Can you provide me with your script and data? It looks like there's some problem creating the source description file.<br>
The error makes me think that the file is too large, but the SrcLargeFile should have handled this.<br>
<span class="m_-8598619938295430762HOEnZb"><font color="#888888"><br>
--Mary<br>
</font></span><div class="m_-8598619938295430762HOEnZb"><div class="m_-8598619938295430762h5"><br>
On Mar 5, 2013, at 4:34 PM, Noel Aloysius wrote:<br>
<br>
> Hi Mary,<br>
><br>
> I am using NCL 6.1.0.<br>
><br>
> I modified the script as per you suggestion, but still get the following error message.<br>
><br>
> It works when I select a smaller region though.<br>
><br>
> ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints<br>
> fatal:NetCDF: Operation not allowed in define mode: error attempting to write variable (grid_dims) to file (/home/fas/saiers/nra23/<wbr>scratch/GPP_NPP_MODIS/GPP_src_<wbr>SCRIP.nc)<br>
> fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 1168 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 1277 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 2976 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 3089 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> fatal:["Execute.c":7741]:<wbr>Execute: Error occurred at or near line 90<br>
><br>
> Thanks,<br>
> Noel<br>
><br>
><br>
><br>
> On Tue, Mar 5, 2013 at 5:25 PM, Mary Haley <<a href="mailto:haley@ucar.edu" target="_blank">haley@ucar.edu</a>> wrote:<br>
> Hi Noel,<br>
><br>
> What version of NCL are you using? It looks like a large file issue, so I think this should fix the problem:<br>
><br>
> Opt@SrcLargeFile = True<br>
> Opt@DstLargeFile = True<br>
><br>
> These are the defaults for Version 6.1.2.<br>
><br>
> --Mary<br>
><br>
> On Mar 5, 2013, at 1:01 PM, Noel Aloysius wrote:<br>
><br>
> > Hi NCL-talk,<br>
> ><br>
> > I want to regrid a high-resolution (~1km) MODIS GPP to a coarse resolution (1x1) grid using ESMF_regrid. The script is appended below.<br>
> > I get the following error messages,<br>
> ><br>
> > ncendef: ncid 131072: NetCDF: One or more variable sizes violate format constraints<br>
> > fatal:NetCDF: Operation not allowed in define mode: error attempting to write variable (grid_dims) to file (/home/fas/scratch/GPP_NPP_<wbr>MODIS/src_SCRIP.nc)<br>
> > fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 1168 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> > fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 1277 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> > fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 2976 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> > fatal:["Execute.c":7743]:<wbr>Execute: Error occurred at or near line 3089 in file $NCARG_ROOT/lib/ncarg/<wbr>nclscripts/esmf/ESMF_<wbr>regridding.ncl<br>
> > fatal:["Execute.c":7741]:<wbr>Execute: Error occurred at or near line 27<br>
> ><br>
> > Thanks in advance for your help to resolve this issue,<br>
> > Noel<br>
> ><br>
> ><br>
> > ;****************************<wbr>******************************<wbr>****************<br>
> > srcDir = "~/"<br>
> > srcFile = "MOD17A3_Science_GPP_2000.nc"<br>
> > srcFileName = str_concat((/srcDir,srcFile/))<br>
> ><br>
> > sfile = addfile(srcFileName,"r") ; source grid<br>
> > gpp = int2flt (sfile->MOD17A3_Science_GPP_<wbr>2000)<br>
> > gpp = gpp(::-1,:) ; South -> North<br>
> ><br>
> > Opt = True<br>
> ><br>
> > Opt@SrcFileName = srcDir + "src_SCRIP.nc"<br>
> > Opt@DstFileName = srcDir + "dst_SCRIP.nc"<br>
> > Opt@WgtFileName = srcDir + "GPP1km_2_halfddeg.nc"<br>
> > Opt@ForceOverwrite = True<br>
> ><br>
> > Opt@DstGridType = "1x1"<br>
> > Opt@DstLLCorner = (/ -59.5, -179.5/)<br>
> > Opt@DstURCorner = (/ 79.5, 179.5/)<br>
> ><br>
> > gpp_regrid = ESMF_regrid(gpp,Opt)<br>
> > ;*****************************<wbr>******************************<wbr>***************<br>
> ><br>
> > printVarSummary(gpp),<br>
> ><br>
> > Variable: gpp<br>
> > Type: float<br>
> > Total Size: 2903040000 bytes<br>
> > 725760000 values<br>
> > Number of Dimensions: 2<br>
> > Dimensions and sizes: [lat | 16800] x [lon | 43200]<br>
> > Coordinates:<br>
> > lat: [-59.99583333334029..79.<wbr>99583333333334]<br>
> > lon: [-179.9958333333333..179.<wbr>9958333333346]<br>
> > Number Of Attributes: 6<br>
> > _FillValue : 1e+20<br>
> > long_name : MOD17A3_Science_GPP_2000<br>
> > esri_pe_string : GEOGCS["GCS_WGS_1984",DATUM["<wbr>D_WGS_1984",SPHEROID["WGS_<wbr>1984",6378137.0,298.257223563]<wbr>],PRIMEM["Greenwich",0.0],<wbr>UNIT["Degree",0.<wbr>0174532925199433]]<br>
> > coordinates : lon lat<br>
> > units : Degree<br>
> > missing_value : 0<br>
> ><br>
> > ncl 12> print(min(gpp))<br>
> > (0) 1<br>
> > ncl 13> print(max(gpp))<br>
> > (0) 65535<br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > ncl-talk mailing list<br>
> > List instructions, subscriber options, unsubscribe:<br>
> > <a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> ncl-talk mailing list<br>
> List instructions, subscriber options, unsubscribe:<br>
> <a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
<br>
</div></div></blockquote></div><br>
</div></div><span><ESMF_regrid_MODIS_Annual_GPP_<wbr>for_NCL_talk.ncl></span>_____________<wbr>______________________________<wbr>____<span class=""><br>ncl-talk mailing list<br>List instructions, subscriber options, unsubscribe:<br><a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br></span></blockquote></div><br></div></div></blockquote></div><br></div>