[ncl-talk] I/O problem with regridding many nc files?

[Archana Dayalu]

Hi there,
I am using the ESMF tools to regrid a bunch of wrf-grid files to
rectangular grid. I am running each month independently (running the script
in 12 independent batches, where each batch corresponds to a month) but the
days in a given month are done serially in that loop. My steps, as a quick
summary, are:
(1) generate the weights once, valid for all variables for all files for
the year. Then, for each of 365 daily temperature and radiation files:
(2) extract the daily T2 (Kelvin), SWDOWN variables which I previously
defined as a 3D array (XDim,YDim,23)
(3) apply the weights to hourly T2, SWDOWN output for a year. This step
involves a loop ... and the relevant part includes:
do hh=0,23

T2_regrid       = ESMF_regrid_with_weights(T2,wgt_file_name,Opt)

SWDOWN_regrid = ESMF_regrid_with_weights(SWDOWN,wgt_file_name,Opt)
T2_array(:,:,hh) = T2_regrid

SWDOWN_array(:,:,hh) = SWDOWN_regrid

; I also have it print out min max for each variable for each hour regridded

end do

(4) Write out the two regridded arrays to a daily netcdf file using method
1, Netcdf4, with file compression level set to 4

The problem: The script fails toward the end (e.g, around the 27th file of
a month) with the following error message and begins to input nonsense
Temperature and radiation values into the array and states the weights file
cannot be opened (it does exist). When I re-run the failed time-period as a
one-time regridding, it regrids just fine. Is ncl unhappy because of the
high I/O? It doesn't fail at the netcdf generation step ... it seems to
fail at the regridding step itself. Thanks for any help! The error output
follows ...

Variable: hh
Type: integer
Total Size: 4 bytes
            1 values
Number of Dimensions: 1
Dimensions and sizes:   [1]
Coordinates:
(0)     15
fatal:["NclNewHDF5.c":3787]:NclNewHDF5.c: Cannot open file:
</n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc>
fatal:["NclAdvancedFile.c":2667]:_NclFileFillHLFS: Could not open
(/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc)
(0)   ESMF_regrid_with_weights: cannot open weight file
'/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc'.
(0)
(0)     TEMP at 2 M: min=239.284   max=299.333
(0)
(0)     min=9.96921e+36   max=9.96921e+36
warning:VarVarWrite: rhs has no dimension name or coordinate variable,
deleting name of lhs dimension number (0) and destroying coordinate var,
 use "(/../)" if this is not desired outcome
warning:VarVarWrite: rhs has no dimension name or coordinate variable,
deleting name of lhs dimension number (1) and destroying coordinate var,
 use "(/../)" if this is not desired outcome
warning:["Execute.c":8578]:Execute: Error occurred at or near line 64 in
file regrid_using_weights_T2SWDOWN_MOD_noplots.ncl

HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
  #000: H5D.c line 461 in H5Dget_space(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type
(...then a series of similar error messages are output .... and then it
continues .... and nonsense is output):

Variable: hh
Type: integer
Total Size: 4 bytes
            1 values
Number of Dimensions: 1
Dimensions and sizes:   [1]
Coordinates:
(0)     17
(0)
(0)     TEMP at 2 M: min=0.0114805   max=299.333
(0)
(0)     DOWNWARD SHORT WAVE FLUX AT GROUND SURFACE: min=0   max=2.90338e-31
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160111/6f723318/attachment.html