[ncl-talk] I/O problem with regridding many nc files?

[Mary Haley]

Are you by any chance using setfileoption to change the write type to
"NetCDF4"?

I'm wondering if there's an issue with going into NetCDF4 mode, that may
cause a problem if you stay in that mode when reading your weights file.

Can you provide us with the full code and all the files needed to run it?
You can use our ftp if the files are not too large:

http://www.ncl.ucar.edu/report_bug.shtml#HowToFTP


--Mary


On Mon, Jan 11, 2016 at 2:09 PM, Archana Dayalu <adayalu at seas.harvard.edu>
wrote:

> Hi there,
> I am using the ESMF tools to regrid a bunch of wrf-grid files to
> rectangular grid. I am running each month independently (running the script
> in 12 independent batches, where each batch corresponds to a month) but the
> days in a given month are done serially in that loop. My steps, as a quick
> summary, are:
> (1) generate the weights once, valid for all variables for all files for
> the year. Then, for each of 365 daily temperature and radiation files:
> (2) extract the daily T2 (Kelvin), SWDOWN variables which I previously
> defined as a 3D array (XDim,YDim,23)
> (3) apply the weights to hourly T2, SWDOWN output for a year. This step
> involves a loop ... and the relevant part includes:
> do hh=0,23
>
> T2_regrid       = ESMF_regrid_with_weights(T2,wgt_file_name,Opt)
>
> SWDOWN_regrid = ESMF_regrid_with_weights(SWDOWN,wgt_file_name,Opt)
> T2_array(:,:,hh) = T2_regrid
>
> SWDOWN_array(:,:,hh) = SWDOWN_regrid
>
> ; I also have it print out min max for each variable for each hour
> regridded
>
> end do
>
> (4) Write out the two regridded arrays to a daily netcdf file using method
> 1, Netcdf4, with file compression level set to 4
>
> The problem: The script fails toward the end (e.g, around the 27th file of
> a month) with the following error message and begins to input nonsense
> Temperature and radiation values into the array and states the weights file
> cannot be opened (it does exist). When I re-run the failed time-period as a
> one-time regridding, it regrids just fine. Is ncl unhappy because of the
> high I/O? It doesn't fail at the netcdf generation step ... it seems to
> fail at the regridding step itself. Thanks for any help! The error output
> follows ...
>
> Variable: hh
> Type: integer
> Total Size: 4 bytes
>             1 values
> Number of Dimensions: 1
> Dimensions and sizes:   [1]
> Coordinates:
> (0)     15
> fatal:["NclNewHDF5.c":3787]:NclNewHDF5.c: Cannot open file:
> </n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc>
> fatal:["NclAdvancedFile.c":2667]:_NclFileFillHLFS: Could not open
> (/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc)
> (0)   ESMF_regrid_with_weights: cannot open weight file
> '/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc'.
> (0)
> (0)     TEMP at 2 M: min=239.284   max=299.333
> (0)
> (0)     min=9.96921e+36   max=9.96921e+36
> warning:VarVarWrite: rhs has no dimension name or coordinate variable,
> deleting name of lhs dimension number (0) and destroying coordinate var,
>  use "(/../)" if this is not desired outcome
> warning:VarVarWrite: rhs has no dimension name or coordinate variable,
> deleting name of lhs dimension number (1) and destroying coordinate var,
>  use "(/../)" if this is not desired outcome
> warning:["Execute.c":8578]:Execute: Error occurred at or near line 64 in
> file regrid_using_weights_T2SWDOWN_MOD_noplots.ncl
>
> HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
>   #000: H5D.c line 461 in H5Dget_space(): not a dataset
>     major: Invalid arguments to routine
>     minor: Inappropriate type
> (...then a series of similar error messages are output .... and then it
> continues .... and nonsense is output):
>
> Variable: hh
> Type: integer
> Total Size: 4 bytes
>             1 values
> Number of Dimensions: 1
> Dimensions and sizes:   [1]
> Coordinates:
> (0)     17
> (0)
> (0)     TEMP at 2 M: min=0.0114805   max=299.333
> (0)
> (0)     DOWNWARD SHORT WAVE FLUX AT GROUND SURFACE: min=0   max=2.90338e-31
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/b6556dde/attachment.html