[ncl-talk] I/O problem with regridding many nc files?

Tue Jan 12 13:04:13 MST 2016

Hi Mary,
I just uploaded my code + data to the ftp site. The files I have uploaded
are:
1) 31 files of type "aux2vprm_d01_2006-01-dd_hrlyavg.nc" (where dd=two
digit day in january), 2.5MB each
2) 1 weights file "T2SWDOWN_to_MODIS_bilinear.faster.nc" (this is kind of
big ... ~6GB)
3) 1 file with the script "regrid_using_weights_T2SWDOWN_MOD_noplots.ncl"
(edit the header of file to specify directories; all else should not
require editing).
Other requested information:
ncl -V = 6.2.1
uname -a = Linux rclogin11.rc.fas.harvard.edu 2.6.32-431.17.1.el6.x86_64 #1
SMP Wed May 7 23:32:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Thanks very much! Please let me know if you have any trouble with the
code/files.
Regards
Archana

On Tue, Jan 12, 2016 at 11:29 AM, Mary Haley <haley at ucar.edu> wrote:

> Are you by any chance using setfileoption to change the write type to
> "NetCDF4"?
>
> I'm wondering if there's an issue with going into NetCDF4 mode, that may
> cause a problem if you stay in that mode when reading your weights file.
>
> Can you provide us with the full code and all the files needed to run it?
> You can use our ftp if the files are not too large:
>
> http://www.ncl.ucar.edu/report_bug.shtml#HowToFTP
>
>
> --Mary
>
>
> On Mon, Jan 11, 2016 at 2:09 PM, Archana Dayalu <adayalu at seas.harvard.edu>
> wrote:
>
>> Hi there,
>> I am using the ESMF tools to regrid a bunch of wrf-grid files to
>> rectangular grid. I am running each month independently (running the script
>> in 12 independent batches, where each batch corresponds to a month) but the
>> days in a given month are done serially in that loop. My steps, as a quick
>> summary, are:
>> (1) generate the weights once, valid for all variables for all files for
>> the year. Then, for each of 365 daily temperature and radiation files:
>> (2) extract the daily T2 (Kelvin), SWDOWN variables which I previously
>> defined as a 3D array (XDim,YDim,23)
>> (3) apply the weights to hourly T2, SWDOWN output for a year. This step
>> involves a loop ... and the relevant part includes:
>> do hh=0,23
>>
>> T2_regrid       = ESMF_regrid_with_weights(T2,wgt_file_name,Opt)
>>
>> SWDOWN_regrid = ESMF_regrid_with_weights(SWDOWN,wgt_file_name,Opt)
>> T2_array(:,:,hh) = T2_regrid
>>
>> SWDOWN_array(:,:,hh) = SWDOWN_regrid
>>
>> ; I also have it print out min max for each variable for each hour
>> regridded
>>
>> end do
>>
>> (4) Write out the two regridded arrays to a daily netcdf file using
>> method 1, Netcdf4, with file compression level set to 4
>>
>> The problem: The script fails toward the end (e.g, around the 27th file
>> of a month) with the following error message and begins to input nonsense
>> Temperature and radiation values into the array and states the weights file
>> cannot be opened (it does exist). When I re-run the failed time-period as a
>> one-time regridding, it regrids just fine. Is ncl unhappy because of the
>> high I/O? It doesn't fail at the netcdf generation step ... it seems to
>> fail at the regridding step itself. Thanks for any help! The error output
>> follows ...
>>
>> Variable: hh
>> Type: integer
>> Total Size: 4 bytes
>>             1 values
>> Number of Dimensions: 1
>> Dimensions and sizes:   [1]
>> Coordinates:
>> (0)     15
>> fatal:["NclNewHDF5.c":3787]:NclNewHDF5.c: Cannot open file:
>> </n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc>
>> fatal:["NclAdvancedFile.c":2667]:_NclFileFillHLFS: Could not open
>> (/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc)
>> (0)   ESMF_regrid_with_weights: cannot open weight file
>> '/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc'.
>> (0)
>> (0)     TEMP at 2 M: min=239.284   max=299.333
>> (0)
>> (0)     min=9.96921e+36   max=9.96921e+36
>> warning:VarVarWrite: rhs has no dimension name or coordinate variable,
>> deleting name of lhs dimension number (0) and destroying coordinate var,
>>  use "(/../)" if this is not desired outcome
>> warning:VarVarWrite: rhs has no dimension name or coordinate variable,
>> deleting name of lhs dimension number (1) and destroying coordinate var,
>>  use "(/../)" if this is not desired outcome
>> warning:["Execute.c":8578]:Execute: Error occurred at or near line 64 in
>> file regrid_using_weights_T2SWDOWN_MOD_noplots.ncl
>>
>> HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
>>   #000: H5D.c line 461 in H5Dget_space(): not a dataset
>>     major: Invalid arguments to routine
>>     minor: Inappropriate type
>> (...then a series of similar error messages are output .... and then it
>> continues .... and nonsense is output):
>>
>> Variable: hh
>> Type: integer
>> Total Size: 4 bytes
>>             1 values
>> Number of Dimensions: 1
>> Dimensions and sizes:   [1]
>> Coordinates:
>> (0)     17
>> (0)
>> (0)     TEMP at 2 M: min=0.0114805   max=299.333
>> (0)
>> (0)     DOWNWARD SHORT WAVE FLUX AT GROUND SURFACE: min=0
>> max=2.90338e-31
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>

-- 
____________________________________________________
*Archana Dayalu*
Graduate Student
Dept. of Earth and Planetary Sciences
Harvard University
24 Oxford Street #402
Cambridge, MA 02138
(617) 384-8206
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/71bb609a/attachment.html 