[ncl-talk] ESMF_regrid issue: 'ESMF_RegridWeightGen' was not successful.

Yuhan Rao yrao at terpmail.umd.edu
Tue Aug 1 13:49:11 MDT 2017


​Hi Mary,

Thanks for your help. I think it might be the memory allocation issue on my
computer since the source data is at high resolution. So I tried to run the
scripts on a computing cluster, and it completed successfully.

I used "patch" because when I first ran the script on my computer with
"bilinear" it failed. So I want to try if "patch" could run successfully,
but it also didn't.

Now I have tried both on the cluster and both succeeded. I will used the
"bilinear" one.

Thanks a lot.

Best,
Douglas

Rao, Yuhan (Douglas)
Doctoral Candidate/Graduate Research Assistant
Department of Geographical Sciences
Cooperative Institute for Climate and Satellites-Maryland
University of Maryland
2138 LeFrak Hall, 7251 Preinkert Dr.
College Park, MD 20742
E-mail: yuhan.rao at gmail.com/yrao at terpmail.umd.edu

On Tue, Aug 1, 2017 at 12:35 PM, Mary Haley <haley at ucar.edu> wrote:

> Douglas,
>
> Yesterday I regridded the full 24 timesteps using the "bilinear"
> interpolation method.  It ran successfully to completion.
>
> Today, I decided to try "patch", since this is what your original script
> had.  This takes *much* longer than bilinear, but it did run to completion.
>
> Both scripts created a NetCDF file with the regridded "ndvi" variable.
>
> I wrote a script to plot the data from both regridded files and compare
> with the data from the original NetCDF file.
>
> I'm not sure what could be the problem on your end, but the "bad_alloc"
> might be that your system ran out of memory. I recommend trying the
> "bilinear" method to see if you can get the script to run to completion.
>
> Here's what the various attached files are:
>
> regrid_ndvi_mod.ncl
>
> Script for regridding your data. Change the "interp_method" to either
> "patch" or "bilinear" before running. This should create a NetCDF file with
> the regridded data called either "ndvi3d_1982_r0.75x0.75_patch.nc" or "
> ndvi3d_1982_r0.75x0.75_bilinear.nc"
>
> regrid_xxxx_output.txt
>
> The two output files from running regrid_ndv_mod.ncl with either bilinear
> or patch.
>
> plot_ndvi_compare.ncl
>
> Script for reading data from the original file and the two
> ndvi3d_1982_r0.75x0.75_xxxx.nc files, and creating a series of paneled
> plots, one for each time step.
>
> plot_ndvi_compare.txt
>
> The output from running plot_ndvi_compare.ncl
>
> ndvi_compare.png
>
> One of the 24 images generated by the plotting script.
>
>
> --Mary
>
>
> On Mon, Jul 31, 2017 at 1:30 PM, Yuhan Rao <yrao at terpmail.umd.edu> wrote:
>
>> Thanks Mary!
>>
>> I have tried several times for this code. Only one time it provide some
>> information in the Log file. I have attached the content of the Log file
>> here.
>>
>> ------------------------------------------------------------
>> ----------------------------------------
>>
>> 20170728 103137.390 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 103900.100 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 104239.462 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 104941.512 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 110122.770 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 112721.098 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 112900.452 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 113226.698 ERROR            PET0 ESMCI_Regrid_F.C:424
>> c_esmc_regrid_create() Internal error: Bad condition std::bad_alloc
>> 20170728 113226.701 ERROR            PET0 ESMF_Regrid.F90:321
>> ESMF_RegridStore Internal error: Bad condition - Internal subroutine call
>> returned Error
>> 20170728 113226.701 ERROR            PET0 ESMF_FieldRegrid.F90:1008
>> ESMF_FieldRegridStoreNX Internal error: Bad condition - Internal subroutine
>> call returned Error
>> 20170728 113226.719 ERROR            PET0 ESMF_RegridWeightGen.F90:1152
>> ESMF_RegridWeightGenFile Internal error: Bad condition - Internal
>> subroutine call returned Error
>> 20170728 113639.935 INFO             PET0 Running with ESMF Version
>> 6.3.0rp1
>>
>> ------------------------------------------------------------
>> ------------------------------------------
>>
>> Thanks for the catch of type mismatch.
>>
>> Best,
>> Douglas
>>
>>>> --
>> Rao, Yuhan (Douglas)
>> Doctoral Candidate/Graduate Research Assistant
>> Department of Geographical Sciences
>> Cooperative Institute for Climate and Satellites-Maryland
>> University of Maryland
>> 2138 LeFrak Hall, 7251 Preinkert Dr.
>> College Park, MD 20742
>> E-mail: yuhan.rao at gmail.com/yrao at terpmail.umd.edu
>>
>> On Mon, Jul 31, 2017 at 1:23 PM, Mary Haley <haley at ucar.edu> wrote:
>>
>>> I ran your script over the first four timesteps instead of all of them,
>>> just so I could see the end result faster.
>>>
>>> It completed without any errors.  I'm now trying it with the full ntime
>>> timesteps. A side note: I had to change:
>>>
>>>  ndvi_all at time          = systemfunc("date")
>>>
>>> to:
>>>
>>>  ndvi_all at time          := systemfunc("date")
>>>
>>> because otherwise there's an error about the attribute being a different
>>> type.
>>>
>>> Meanwhile, can you send me the PET0.RegridWeightGen.Log file if you
>>> still have it?
>>> ​ It will hopefully explain what was causing your error.
>>>
>>> --Mary
>>>
>>>>>>
>>> On Mon, Jul 31, 2017 at 9:40 AM, Mary Haley <haley at ucar.edu> wrote:
>>>
>>>> Hi Douglas,
>>>>
>>>> Thanks for providing a clean script and your data.  I tried running it
>>>> on Friday, but it was taking a really long time for some reason and I never
>>>> got back to it.
>>>>
>>>> I'm running it again to see if I can reproduce the error.
>>>>
>>>> --Mary
>>>>
>>>>
>>>> On Fri, Jul 28, 2017 at 11:54 AM, Yuhan Rao <yrao at terpmail.umd.edu>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> My apologies for the previous incomplete message.
>>>>>
>>>>> I was running into issues of ESMF_regid_with_weights. I am trying to
>>>>> regrid the the NDVI data (at the resolution of 1/12 degree) to a 0.75
>>>>> degree for my analysis.
>>>>> When I try to regrid them using ESMF_regrid_with_weights functions, I
>>>>> ran into the problem during weight generation step. The information
>>>>> provided by the debug option is listed here:
>>>>> (0) =====> CPU Elapsed Time: rectilinear_to_SCRIP: 9.34327 seconds
>>>>> <=====
>>>>> (0) ESMF_regrid_gen_weights: number of processors used: 1
>>>>> (0) --------------------------------------------------
>>>>> (0) ESMF_regrid_gen_weights: the following command is about to be
>>>>> executed on the system:
>>>>> (0) 'ESMF_RegridWeightGen --source src_SCRIP.nc --destination
>>>>> dst_SCRIP.nc --weight gimms_to_rect.nc --method patch -i --check'
>>>>> (0) --------------------------------------------------
>>>>> (0) ESMF_regrid_gen_weights: output from 'ESMF_RegridWeightGen':
>>>>> (0)      Starting weight generation with these inputs:
>>>>> (1)        Source File: src_SCRIP.nc
>>>>> (2)        Destination File: dst_SCRIP.nc
>>>>> (3)        Weight File: gimms_to_rect.nc
>>>>> (4)        Source File is in SCRIP format
>>>>> (5)        Source Grid is a global grid
>>>>> (6)        Source Grid is a logically rectangular grid
>>>>> (7)        Destination File is in SCRIP format
>>>>> (8)        Destination Grid is a global grid
>>>>> (9)        Destination Grid is a logically rectangular grid
>>>>> (10)        Regrid Method: patch
>>>>> (11)        Pole option: ALL
>>>>> (12)        Ignore unmapped destination points
>>>>> (13)        Norm Type: dstarea
>>>>> (14)
>>>>> (0) --------------------------------------------------
>>>>> (0) ESMF_regrid_gen_weights: 'ESMF_RegridWeightGen' was not
>>>>> successful.
>>>>>
>>>>> My scripts is listed below:
>>>>>
>>>>> ;**********************************************
>>>>> ;  process_ndvi3g_data.ncl
>>>>> ;
>>>>> ;    -read in ndvi 3g data
>>>>> ;    -regrid 1/12 degree to 0.75 deg
>>>>> ;    -during regridding, mask climatology value
>>>>> ;***********************************************
>>>>>
>>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl"
>>>>>
>>>>> pixel   = 0.75                           ; new grid size
>>>>> latN    = todouble(  90.-0.5*pixel)      ; geolocation limits
>>>>> latS    = todouble( -90.+0.5*pixel)      ; for center of coner grids
>>>>> lonW    = todouble(-180.+0.5*pixel)
>>>>> lonE    = todouble( 180.-0.5*pixel)
>>>>> nlon    = toint((lonE-lonW)/pixel+1)
>>>>> nlat    = toint((latN-latS)/pixel+1)
>>>>>
>>>>> lat_new = todouble(fspan(latS,latN,nlat))
>>>>> lon_new = todouble(fspan(lonW,lonE,nlon))
>>>>>
>>>>> lat_new!0             = "lat"
>>>>> lat_new&lat           = lat_new
>>>>> lat_new&lat at long_name = "latitude"
>>>>> lat_new&lat at units     = "degrees_north"
>>>>> lon_new!0             = "lon"
>>>>> lon_new&lon           = lon_new
>>>>> lon_new&lon at long_name = "longitude"
>>>>> lon_new&lon at units     = "degrees_east"
>>>>>
>>>>> ndvi_all = new((/24,nlat,nlon/),float)
>>>>>
>>>>> fin  = addfile("ndvi3g_1982_nasa.nc","r")
>>>>> ndvi = fin->ndvi
>>>>> flag = fin->flag
>>>>> printVarSummary(ndvi)
>>>>>
>>>>> ntime = dimsizes(ndvi&time)
>>>>> ;---iterate through each 15-day period
>>>>> ;   masking out data with flag=2 & missing data
>>>>>
>>>>> ;---Generate dst grid file
>>>>>
>>>>>   dstGridName        = "dst_SCRIP.nc"
>>>>>
>>>>>   Opt                = True
>>>>>   Opt at ForceOverwrite = True
>>>>>   Opt at PrintTimins    = True
>>>>>
>>>>>   rectilinear_to_SCRIP(dstGridName,lat_new,lon_new,Opt)
>>>>>   delete(Opt)
>>>>>
>>>>>   do itime = 0, ntime-1
>>>>>
>>>>> ;---Generate Src grids (mask is changing)
>>>>>    tmpndvi = ndvi(itime,:,:)
>>>>>    tmpmask = flag(itime,:,:)
>>>>>    tmpmask = where((.not.ismissing(tmpndvi)).and.(tmpmask.ne.2),1,0)
>>>>>
>>>>>    srcGridName        = "src_SCRIP.nc"
>>>>>    wgtFileName        = "gimms_to_rect.nc"
>>>>>
>>>>>    Opt                = True
>>>>>    Opt at GridMask       = tmpmask
>>>>>    Opt at ForceOverwrite = True
>>>>>    Opt at PrintTimings   = True
>>>>>    Opt at Title          = "GIMMS Grid"
>>>>>
>>>>>    rectilinear_to_SCRIP(srcGridName,tmpndvi&lat,tmpndvi&lon,Opt)
>>>>>    delete(Opt)
>>>>>
>>>>> ;----------------------------------------------
>>>>> ;  Generate the weights between Src to Dst grids
>>>>> ;----------------------------------------------
>>>>>    Opt                = True
>>>>>    Opt at InterpMethod   = "patch"
>>>>>    Opt at ForceOverwrite = True
>>>>>    Opt at PrintTimings   = True
>>>>>    Opt at Debug          = True
>>>>>    Opt at Check          = True
>>>>> ;   setfileoption("nc","Format","LargeFile")
>>>>>    ESMF_regrid_gen_weights(srcGridName,dstGridName,wgtFileName,Opt)
>>>>>    delete(Opt)
>>>>>
>>>>>    ;--------------------------------------------
>>>>>    ; esmf_regrid with precalculated weight file
>>>>>    ;--------------------------------------------
>>>>>    Opt                 = True
>>>>>    Opt at PrintTimings    = True
>>>>>
>>>>>    tmp_regrid = ESMF_regrid_with_weights(tmpndvi,wgtFileName,Opt)
>>>>>    printVarSummary(tmp_regrid)
>>>>>
>>>>>    ndvi_all(itime,:,:) = tmp_regrid
>>>>>
>>>>>   end do ;itime
>>>>>
>>>>> ndvi_all!0             = "time"
>>>>> ndvi_all!1             = "lat"
>>>>> ndvi_all!2             = "lon"
>>>>> ndvi_all&lat           = lat_new
>>>>> ndvi_all&lon           = lon_new
>>>>> ndvi_all&lat at units     = "degrees_north"
>>>>> ndvi_all&lon at units     = "degrees_east"
>>>>> ndvi_all&lat at long_name = "latitude"
>>>>> ndvi_all&lon at long_name = "longitude"
>>>>>
>>>>> ndvi_all at time          = systemfunc("date")
>>>>> ndvi_all at comments      = "Regridded GIMMS 3g NDVI data from NASA NEX
>>>>> using ESMF_regrid in NCL. Only NDVI values with flag (0,1) is used in
>>>>> regridding."
>>>>> ndvi_all at long_name     = "Normalized Difference Vegetation Index
>>>>> (GIMMS 3g)"
>>>>> ndvi_all at scale         = 0.0001
>>>>>
>>>>> ndvi_all               = where(ismissing(ndvi_all),tosh
>>>>> ort(-32768),toshort(ndvi_all*10000))
>>>>> ndvi_all at _FillValue   := toshort(-32768)    ; conver to short to
>>>>> compress
>>>>>
>>>>> ndvi_all&time          = fspan(0.5,12,24)
>>>>>
>>>>> oname = "ndvi3d_1982_r0.75x0.75.nc"
>>>>> system("rm -f "+oname)
>>>>> fout  = addfile(oname,"c")
>>>>> fout->ndvi = ndvi_all
>>>>> ------------------------------------------------------------
>>>>> ---------------------------------------------------------
>>>>>
>>>>> I have put my data in the ftp.cgd.ucar.edu/incoming, the file name is
>>>>> "ndvi3g_1982_nasa.nc".
>>>>>
>>>>> Could you help me look into this issue?
>>>>>
>>>>> Thanks!
>>>>> Best,
>>>>> Douglas
>>>>>
>>>>>>>>>> --
>>>>> Rao, Yuhan (Douglas)
>>>>> Doctoral Candidate/Graduate Research Assistant
>>>>> Department of Geographical Sciences
>>>>> Cooperative Institute for Climate and Satellites-Maryland
>>>>> University of Maryland
>>>>> 2138 LeFrak Hall, 7251 Preinkert Dr.
>>>>> College Park, MD 20742
>>>>> E-mail: yuhan.rao at gmail.com/yrao at terpmail.umd.edu
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170801/865c9ad8/attachment.html 


More information about the ncl-talk mailing list