[ncl-talk] ESMF_regrid issue: 'ESMF_RegridWeightGen' was not successful.
Yuhan Rao
yrao at terpmail.umd.edu
Tue Aug 1 13:49:11 MDT 2017
Hi Mary,
Thanks for your help. I think it might be the memory allocation issue on my
computer since the source data is at high resolution. So I tried to run the
scripts on a computing cluster, and it completed successfully.
I used "patch" because when I first ran the script on my computer with
"bilinear" it failed. So I want to try if "patch" could run successfully,
but it also didn't.
Now I have tried both on the cluster and both succeeded. I will used the
"bilinear" one.
Thanks a lot.
Best,
Douglas
Rao, Yuhan (Douglas)
Doctoral Candidate/Graduate Research Assistant
Department of Geographical Sciences
Cooperative Institute for Climate and Satellites-Maryland
University of Maryland
2138 LeFrak Hall, 7251 Preinkert Dr.
College Park, MD 20742
E-mail: yuhan.rao at gmail.com/yrao at terpmail.umd.edu
On Tue, Aug 1, 2017 at 12:35 PM, Mary Haley <haley at ucar.edu> wrote:
> Douglas,
>
> Yesterday I regridded the full 24 timesteps using the "bilinear"
> interpolation method. It ran successfully to completion.
>
> Today, I decided to try "patch", since this is what your original script
> had. This takes *much* longer than bilinear, but it did run to completion.
>
> Both scripts created a NetCDF file with the regridded "ndvi" variable.
>
> I wrote a script to plot the data from both regridded files and compare
> with the data from the original NetCDF file.
>
> I'm not sure what could be the problem on your end, but the "bad_alloc"
> might be that your system ran out of memory. I recommend trying the
> "bilinear" method to see if you can get the script to run to completion.
>
> Here's what the various attached files are:
>
> regrid_ndvi_mod.ncl
>
> Script for regridding your data. Change the "interp_method" to either
> "patch" or "bilinear" before running. This should create a NetCDF file with
> the regridded data called either "ndvi3d_1982_r0.75x0.75_patch.nc" or "
> ndvi3d_1982_r0.75x0.75_bilinear.nc"
>
> regrid_xxxx_output.txt
>
> The two output files from running regrid_ndv_mod.ncl with either bilinear
> or patch.
>
> plot_ndvi_compare.ncl
>
> Script for reading data from the original file and the two
> ndvi3d_1982_r0.75x0.75_xxxx.nc files, and creating a series of paneled
> plots, one for each time step.
>
> plot_ndvi_compare.txt
>
> The output from running plot_ndvi_compare.ncl
>
> ndvi_compare.png
>
> One of the 24 images generated by the plotting script.
>
>
> --Mary
>
>
> On Mon, Jul 31, 2017 at 1:30 PM, Yuhan Rao <yrao at terpmail.umd.edu> wrote:
>
>> Thanks Mary!
>>
>> I have tried several times for this code. Only one time it provide some
>> information in the Log file. I have attached the content of the Log file
>> here.
>>
>> ------------------------------------------------------------
>> ----------------------------------------
>>
>> 20170728 103137.390 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 103900.100 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 104239.462 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 104941.512 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 110122.770 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 112721.098 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 112900.452 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>> 20170728 113226.698 ERROR PET0 ESMCI_Regrid_F.C:424
>> c_esmc_regrid_create() Internal error: Bad condition std::bad_alloc
>> 20170728 113226.701 ERROR PET0 ESMF_Regrid.F90:321
>> ESMF_RegridStore Internal error: Bad condition - Internal subroutine call
>> returned Error
>> 20170728 113226.701 ERROR PET0 ESMF_FieldRegrid.F90:1008
>> ESMF_FieldRegridStoreNX Internal error: Bad condition - Internal subroutine
>> call returned Error
>> 20170728 113226.719 ERROR PET0 ESMF_RegridWeightGen.F90:1152
>> ESMF_RegridWeightGenFile Internal error: Bad condition - Internal
>> subroutine call returned Error
>> 20170728 113639.935 INFO PET0 Running with ESMF Version
>> 6.3.0rp1
>>
>> ------------------------------------------------------------
>> ------------------------------------------
>>
>> Thanks for the catch of type mismatch.
>>
>> Best,
>> Douglas
>>
>>
>> --
>> Rao, Yuhan (Douglas)
>> Doctoral Candidate/Graduate Research Assistant
>> Department of Geographical Sciences
>> Cooperative Institute for Climate and Satellites-Maryland
>> University of Maryland
>> 2138 LeFrak Hall, 7251 Preinkert Dr.
>> College Park, MD 20742
>> E-mail: yuhan.rao at gmail.com/yrao at terpmail.umd.edu
>>
>> On Mon, Jul 31, 2017 at 1:23 PM, Mary Haley <haley at ucar.edu> wrote:
>>
>>> I ran your script over the first four timesteps instead of all of them,
>>> just so I could see the end result faster.
>>>
>>> It completed without any errors. I'm now trying it with the full ntime
>>> timesteps. A side note: I had to change:
>>>
>>> ndvi_all at time = systemfunc("date")
>>>
>>> to:
>>>
>>> ndvi_all at time := systemfunc("date")
>>>
>>> because otherwise there's an error about the attribute being a different
>>> type.
>>>
>>> Meanwhile, can you send me the PET0.RegridWeightGen.Log file if you
>>> still have it?
>>> It will hopefully explain what was causing your error.
>>>
>>> --Mary
>>>
>>>
>>>
>>> On Mon, Jul 31, 2017 at 9:40 AM, Mary Haley <haley at ucar.edu> wrote:
>>>
>>>> Hi Douglas,
>>>>
>>>> Thanks for providing a clean script and your data. I tried running it
>>>> on Friday, but it was taking a really long time for some reason and I never
>>>> got back to it.
>>>>
>>>> I'm running it again to see if I can reproduce the error.
>>>>
>>>> --Mary
>>>>
>>>>
>>>> On Fri, Jul 28, 2017 at 11:54 AM, Yuhan Rao <yrao at terpmail.umd.edu>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> My apologies for the previous incomplete message.
>>>>>
>>>>> I was running into issues of ESMF_regid_with_weights. I am trying to
>>>>> regrid the the NDVI data (at the resolution of 1/12 degree) to a 0.75
>>>>> degree for my analysis.
>>>>> When I try to regrid them using ESMF_regrid_with_weights functions, I
>>>>> ran into the problem during weight generation step. The information
>>>>> provided by the debug option is listed here:
>>>>> (0) =====> CPU Elapsed Time: rectilinear_to_SCRIP: 9.34327 seconds
>>>>> <=====
>>>>> (0) ESMF_regrid_gen_weights: number of processors used: 1
>>>>> (0) --------------------------------------------------
>>>>> (0) ESMF_regrid_gen_weights: the following command is about to be
>>>>> executed on the system:
>>>>> (0) 'ESMF_RegridWeightGen --source src_SCRIP.nc --destination
>>>>> dst_SCRIP.nc --weight gimms_to_rect.nc --method patch -i --check'
>>>>> (0) --------------------------------------------------
>>>>> (0) ESMF_regrid_gen_weights: output from 'ESMF_RegridWeightGen':
>>>>> (0) Starting weight generation with these inputs:
>>>>> (1) Source File: src_SCRIP.nc
>>>>> (2) Destination File: dst_SCRIP.nc
>>>>> (3) Weight File: gimms_to_rect.nc
>>>>> (4) Source File is in SCRIP format
>>>>> (5) Source Grid is a global grid
>>>>> (6) Source Grid is a logically rectangular grid
>>>>> (7) Destination File is in SCRIP format
>>>>> (8) Destination Grid is a global grid
>>>>> (9) Destination Grid is a logically rectangular grid
>>>>> (10) Regrid Method: patch
>>>>> (11) Pole option: ALL
>>>>> (12) Ignore unmapped destination points
>>>>> (13) Norm Type: dstarea
>>>>> (14)
>>>>> (0) --------------------------------------------------
>>>>> (0) ESMF_regrid_gen_weights: 'ESMF_RegridWeightGen' was not
>>>>> successful.
>>>>>
>>>>> My scripts is listed below:
>>>>>
>>>>> ;**********************************************
>>>>> ; process_ndvi3g_data.ncl
>>>>> ;
>>>>> ; -read in ndvi 3g data
>>>>> ; -regrid 1/12 degree to 0.75 deg
>>>>> ; -during regridding, mask climatology value
>>>>> ;***********************************************
>>>>>
>>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl"
>>>>>
>>>>> pixel = 0.75 ; new grid size
>>>>> latN = todouble( 90.-0.5*pixel) ; geolocation limits
>>>>> latS = todouble( -90.+0.5*pixel) ; for center of coner grids
>>>>> lonW = todouble(-180.+0.5*pixel)
>>>>> lonE = todouble( 180.-0.5*pixel)
>>>>> nlon = toint((lonE-lonW)/pixel+1)
>>>>> nlat = toint((latN-latS)/pixel+1)
>>>>>
>>>>> lat_new = todouble(fspan(latS,latN,nlat))
>>>>> lon_new = todouble(fspan(lonW,lonE,nlon))
>>>>>
>>>>> lat_new!0 = "lat"
>>>>> lat_new&lat = lat_new
>>>>> lat_new&lat at long_name = "latitude"
>>>>> lat_new&lat at units = "degrees_north"
>>>>> lon_new!0 = "lon"
>>>>> lon_new&lon = lon_new
>>>>> lon_new&lon at long_name = "longitude"
>>>>> lon_new&lon at units = "degrees_east"
>>>>>
>>>>> ndvi_all = new((/24,nlat,nlon/),float)
>>>>>
>>>>> fin = addfile("ndvi3g_1982_nasa.nc","r")
>>>>> ndvi = fin->ndvi
>>>>> flag = fin->flag
>>>>> printVarSummary(ndvi)
>>>>>
>>>>> ntime = dimsizes(ndvi&time)
>>>>> ;---iterate through each 15-day period
>>>>> ; masking out data with flag=2 & missing data
>>>>>
>>>>> ;---Generate dst grid file
>>>>>
>>>>> dstGridName = "dst_SCRIP.nc"
>>>>>
>>>>> Opt = True
>>>>> Opt at ForceOverwrite = True
>>>>> Opt at PrintTimins = True
>>>>>
>>>>> rectilinear_to_SCRIP(dstGridName,lat_new,lon_new,Opt)
>>>>> delete(Opt)
>>>>>
>>>>> do itime = 0, ntime-1
>>>>>
>>>>> ;---Generate Src grids (mask is changing)
>>>>> tmpndvi = ndvi(itime,:,:)
>>>>> tmpmask = flag(itime,:,:)
>>>>> tmpmask = where((.not.ismissing(tmpndvi)).and.(tmpmask.ne.2),1,0)
>>>>>
>>>>> srcGridName = "src_SCRIP.nc"
>>>>> wgtFileName = "gimms_to_rect.nc"
>>>>>
>>>>> Opt = True
>>>>> Opt at GridMask = tmpmask
>>>>> Opt at ForceOverwrite = True
>>>>> Opt at PrintTimings = True
>>>>> Opt at Title = "GIMMS Grid"
>>>>>
>>>>> rectilinear_to_SCRIP(srcGridName,tmpndvi&lat,tmpndvi&lon,Opt)
>>>>> delete(Opt)
>>>>>
>>>>> ;----------------------------------------------
>>>>> ; Generate the weights between Src to Dst grids
>>>>> ;----------------------------------------------
>>>>> Opt = True
>>>>> Opt at InterpMethod = "patch"
>>>>> Opt at ForceOverwrite = True
>>>>> Opt at PrintTimings = True
>>>>> Opt at Debug = True
>>>>> Opt at Check = True
>>>>> ; setfileoption("nc","Format","LargeFile")
>>>>> ESMF_regrid_gen_weights(srcGridName,dstGridName,wgtFileName,Opt)
>>>>> delete(Opt)
>>>>>
>>>>> ;--------------------------------------------
>>>>> ; esmf_regrid with precalculated weight file
>>>>> ;--------------------------------------------
>>>>> Opt = True
>>>>> Opt at PrintTimings = True
>>>>>
>>>>> tmp_regrid = ESMF_regrid_with_weights(tmpndvi,wgtFileName,Opt)
>>>>> printVarSummary(tmp_regrid)
>>>>>
>>>>> ndvi_all(itime,:,:) = tmp_regrid
>>>>>
>>>>> end do ;itime
>>>>>
>>>>> ndvi_all!0 = "time"
>>>>> ndvi_all!1 = "lat"
>>>>> ndvi_all!2 = "lon"
>>>>> ndvi_all&lat = lat_new
>>>>> ndvi_all&lon = lon_new
>>>>> ndvi_all&lat at units = "degrees_north"
>>>>> ndvi_all&lon at units = "degrees_east"
>>>>> ndvi_all&lat at long_name = "latitude"
>>>>> ndvi_all&lon at long_name = "longitude"
>>>>>
>>>>> ndvi_all at time = systemfunc("date")
>>>>> ndvi_all at comments = "Regridded GIMMS 3g NDVI data from NASA NEX
>>>>> using ESMF_regrid in NCL. Only NDVI values with flag (0,1) is used in
>>>>> regridding."
>>>>> ndvi_all at long_name = "Normalized Difference Vegetation Index
>>>>> (GIMMS 3g)"
>>>>> ndvi_all at scale = 0.0001
>>>>>
>>>>> ndvi_all = where(ismissing(ndvi_all),tosh
>>>>> ort(-32768),toshort(ndvi_all*10000))
>>>>> ndvi_all at _FillValue := toshort(-32768) ; conver to short to
>>>>> compress
>>>>>
>>>>> ndvi_all&time = fspan(0.5,12,24)
>>>>>
>>>>> oname = "ndvi3d_1982_r0.75x0.75.nc"
>>>>> system("rm -f "+oname)
>>>>> fout = addfile(oname,"c")
>>>>> fout->ndvi = ndvi_all
>>>>> ------------------------------------------------------------
>>>>> ---------------------------------------------------------
>>>>>
>>>>> I have put my data in the ftp.cgd.ucar.edu/incoming, the file name is
>>>>> "ndvi3g_1982_nasa.nc".
>>>>>
>>>>> Could you help me look into this issue?
>>>>>
>>>>> Thanks!
>>>>> Best,
>>>>> Douglas
>>>>>
>>>>>
>>>>> --
>>>>> Rao, Yuhan (Douglas)
>>>>> Doctoral Candidate/Graduate Research Assistant
>>>>> Department of Geographical Sciences
>>>>> Cooperative Institute for Climate and Satellites-Maryland
>>>>> University of Maryland
>>>>> 2138 LeFrak Hall, 7251 Preinkert Dr.
>>>>> College Park, MD 20742
>>>>> E-mail: yuhan.rao at gmail.com/yrao at terpmail.umd.edu
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170801/865c9ad8/attachment.html
More information about the ncl-talk
mailing list