[ncl-talk] ESMF regrid
Mary Haley
haley at ucar.edu
Fri Feb 17 13:07:11 MST 2017
Hi all,
I'm responding to this old ESMF question at the request of somebody who
found this thread. but not a satisfactory answer.
If you are trying to regrid really large variables using the NCL-ESMF
regridding suite, then you may need to set one of the the "NetCDFType"
options to "netcdf4" which allows you to write > than 4 GB to a file.
For example, if your source grid is really large, and ESMF_regrid gives you
an error:
ncendef: ncid 131072: NetCDF: One or more variable sizes violate format
constraints
then try setting:
Opt at SrcNetCDFType = "netcdf4"
There are similar options for the destination file "DstNetCDFType" and
weights file "WgtNetCDFType".
Noel, my hope is that you found this out long before now!
--Mary
On Tue, Mar 19, 2013 at 8:05 AM, Mary Haley <haley at ucar.edu> wrote:
>
> Hi Noel,
>
> I see what the problem is, and I'm not sure yet how to fix it.
>
> The issue is in order to do ESMF regridding, we need to write an
> intermediate NetCDF SCRIP file. This file will have two double variables of
> size grid_size, which is equal to nlat * nlon.
>
> In your case:
>
> nlat/nlon = 16,800/43,200
> grid_size = nlat*lon = 725760000
> variable to be written to the file = 725760000*8 = 5,806,080,000 bytes
>
> This is > 4 GB. This violates a NetCDF constraint that individual
> variables be less than 4 GB.
>
> My first thought is that if you can create a NetCDF-4 file, then you won't
> be limited by this constraint. The problem is that the ESMF regridding tool
> we use appears to have a constraint that the intermediate NetCDF files be
> NetCDF-3 (classic or 64-bit offset) files, and not NetCDF-4.
>
> I will check into this and get back to you.
>
> --Mary
>
>
> On Mar 7, 2013, at 10:27 AM, Noel Aloysius wrote:
>
> Hi Mary,
>
> The data file is uploaded to the ftp server - ftp://ftp.cgd.ucar.edu/
> incoming/MOD17A3_Science_GPP_2001.zip.
>
> The script is attached.
>
> I want to regrid ftom 1km to 0.5x0.5deg lat/lon.
>
> Thanks,
> Noel
>
>
> Noel Aloysius
>
>
> On Thu, Mar 7, 2013 at 9:56 AM, Mary Haley <haley at ucar.edu> wrote:
>
>> Hi Noel,
>>
>> Can you provide me with your script and data? It looks like there's some
>> problem creating the source description file.
>> The error makes me think that the file is too large, but the SrcLargeFile
>> should have handled this.
>>
>> --Mary
>>
>> On Mar 5, 2013, at 4:34 PM, Noel Aloysius wrote:
>>
>> > Hi Mary,
>> >
>> > I am using NCL 6.1.0.
>> >
>> > I modified the script as per you suggestion, but still get the
>> following error message.
>> >
>> > It works when I select a smaller region though.
>> >
>> > ncendef: ncid 131072: NetCDF: One or more variable sizes violate format
>> constraints
>> > fatal:NetCDF: Operation not allowed in define mode: error attempting to
>> write variable (grid_dims) to file (/home/fas/saiers/nra23/
>> scratch/GPP_NPP_MODIS/GPP_src_SCRIP.nc)
>> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 1168
>> in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 1277
>> in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 2976
>> in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > fatal:["Execute.c":7743]:Execute: Error occurred at or near line 3089
>> in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > fatal:["Execute.c":7741]:Execute: Error occurred at or near line 90
>> >
>> > Thanks,
>> > Noel
>> >
>> >
>> >
>> > On Tue, Mar 5, 2013 at 5:25 PM, Mary Haley <haley at ucar.edu> wrote:
>> > Hi Noel,
>> >
>> > What version of NCL are you using? It looks like a large file issue,
>> so I think this should fix the problem:
>> >
>> > Opt at SrcLargeFile = True
>> > Opt at DstLargeFile = True
>> >
>> > These are the defaults for Version 6.1.2.
>> >
>> > --Mary
>> >
>> > On Mar 5, 2013, at 1:01 PM, Noel Aloysius wrote:
>> >
>> > > Hi NCL-talk,
>> > >
>> > > I want to regrid a high-resolution (~1km) MODIS GPP to a coarse
>> resolution (1x1) grid using ESMF_regrid. The script is appended below.
>> > > I get the following error messages,
>> > >
>> > > ncendef: ncid 131072: NetCDF: One or more variable sizes violate
>> format constraints
>> > > fatal:NetCDF: Operation not allowed in define mode: error attempting
>> to write variable (grid_dims) to file (/home/fas/scratch/GPP_NPP_
>> MODIS/src_SCRIP.nc)
>> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line
>> 1168 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line
>> 1277 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line
>> 2976 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > > fatal:["Execute.c":7743]:Execute: Error occurred at or near line
>> 3089 in file $NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl
>> > > fatal:["Execute.c":7741]:Execute: Error occurred at or near line 27
>> > >
>> > > Thanks in advance for your help to resolve this issue,
>> > > Noel
>> > >
>> > >
>> > > ;**********************************************************
>> ****************
>> > > srcDir = "~/"
>> > > srcFile = "MOD17A3_Science_GPP_2000.nc"
>> > > srcFileName = str_concat((/srcDir,srcFile/))
>> > >
>> > > sfile = addfile(srcFileName,"r") ; source grid
>> > > gpp = int2flt (sfile->MOD17A3_Science_GPP_2000)
>> > > gpp = gpp(::-1,:) ;
>> South -> North
>> > >
>> > > Opt = True
>> > >
>> > > Opt at SrcFileName = srcDir + "src_SCRIP.nc"
>> > > Opt at DstFileName = srcDir + "dst_SCRIP.nc"
>> > > Opt at WgtFileName = srcDir + "GPP1km_2_halfddeg.nc"
>> > > Opt at ForceOverwrite = True
>> > >
>> > > Opt at DstGridType = "1x1"
>> > > Opt at DstLLCorner = (/ -59.5, -179.5/)
>> > > Opt at DstURCorner = (/ 79.5, 179.5/)
>> > >
>> > > gpp_regrid = ESMF_regrid(gpp,Opt)
>> > > ;***********************************************************
>> ***************
>> > >
>> > > printVarSummary(gpp),
>> > >
>> > > Variable: gpp
>> > > Type: float
>> > > Total Size: 2903040000 bytes
>> > > 725760000 values
>> > > Number of Dimensions: 2
>> > > Dimensions and sizes: [lat | 16800] x [lon | 43200]
>> > > Coordinates:
>> > > lat: [-59.99583333334029..79.99583333333334]
>> > > lon: [-179.9958333333333..179.9958333333346]
>> > > Number Of Attributes: 6
>> > > _FillValue : 1e+20
>> > > long_name : MOD17A3_Science_GPP_2000
>> > > esri_pe_string : GEOGCS["GCS_WGS_1984",DATUM["
>> D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]
>> ],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]
>> > > coordinates : lon lat
>> > > units : Degree
>> > > missing_value : 0
>> > >
>> > > ncl 12> print(min(gpp))
>> > > (0) 1
>> > > ncl 13> print(max(gpp))
>> > > (0) 65535
>> > >
>> > >
>> > > _______________________________________________
>> > > ncl-talk mailing list
>> > > List instructions, subscriber options, unsubscribe:
>> > > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> >
>> >
>> > _______________________________________________
>> > ncl-talk mailing list
>> > List instructions, subscriber options, unsubscribe:
>> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
> <ESMF_regrid_MODIS_Annual_GPP_for_NCL_talk.ncl>_____________
> __________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170217/6b83dd31/attachment.html
More information about the ncl-talk
mailing list