[ncl-talk] I/O problem with regridding many nc files?

Tue Jan 12 13:18:05 MST 2016

Sorry, a couple more things:
1) To answer your question, Mary: I am using the setfileoption command to
change file type.
2) I just re-ran the whole script a different way (i.e., I broke it up by
common days of a month rather than entire months ... so instead of 12
independent loops of 28-31 iterations, I did 31 independent loops of 4-12
iterations). So far it seems to be completing successfully which indicates
that it was a problem with too many calls to the ESMF regrid with weights
function? But I don't understand why that would be so (if it were the
issue).
Regards
Archana

On Tue, Jan 12, 2016 at 3:04 PM, Archana Dayalu <adayalu at seas.harvard.edu>
wrote:

> Hi Mary,
> I just uploaded my code + data to the ftp site. The files I have uploaded
> are:
> 1) 31 files of type "aux2vprm_d01_2006-01-dd_hrlyavg.nc" (where dd=two
> digit day in january), 2.5MB each
> 2) 1 weights file "T2SWDOWN_to_MODIS_bilinear.faster.nc" (this is kind of
> big ... ~6GB)
> 3) 1 file with the script "regrid_using_weights_T2SWDOWN_MOD_noplots.ncl"
> (edit the header of file to specify directories; all else should not
> require editing).
> Other requested information:
> ncl -V = 6.2.1
> uname -a = Linux rclogin11.rc.fas.harvard.edu 2.6.32-431.17.1.el6.x86_64
> #1 SMP Wed May 7 23:32:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> Thanks very much! Please let me know if you have any trouble with the
> code/files.
> Regards
> Archana
>
> On Tue, Jan 12, 2016 at 11:29 AM, Mary Haley <haley at ucar.edu> wrote:
>
>> Are you by any chance using setfileoption to change the write type to
>> "NetCDF4"?
>>
>> I'm wondering if there's an issue with going into NetCDF4 mode, that may
>> cause a problem if you stay in that mode when reading your weights file.
>>
>> Can you provide us with the full code and all the files needed to run it?
>> You can use our ftp if the files are not too large:
>>
>> http://www.ncl.ucar.edu/report_bug.shtml#HowToFTP
>>
>>
>> --Mary
>>
>>
>> On Mon, Jan 11, 2016 at 2:09 PM, Archana Dayalu <adayalu at seas.harvard.edu
>> > wrote:
>>
>>> Hi there,
>>> I am using the ESMF tools to regrid a bunch of wrf-grid files to
>>> rectangular grid. I am running each month independently (running the script
>>> in 12 independent batches, where each batch corresponds to a month) but the
>>> days in a given month are done serially in that loop. My steps, as a quick
>>> summary, are:
>>> (1) generate the weights once, valid for all variables for all files for
>>> the year. Then, for each of 365 daily temperature and radiation files:
>>> (2) extract the daily T2 (Kelvin), SWDOWN variables which I previously
>>> defined as a 3D array (XDim,YDim,23)
>>> (3) apply the weights to hourly T2, SWDOWN output for a year. This step
>>> involves a loop ... and the relevant part includes:
>>> do hh=0,23
>>>
>>> T2_regrid       = ESMF_regrid_with_weights(T2,wgt_file_name,Opt)
>>>
>>> SWDOWN_regrid = ESMF_regrid_with_weights(SWDOWN,wgt_file_name,Opt)
>>> T2_array(:,:,hh) = T2_regrid
>>>
>>> SWDOWN_array(:,:,hh) = SWDOWN_regrid
>>>
>>> ; I also have it print out min max for each variable for each hour
>>> regridded
>>>
>>> end do
>>>
>>> (4) Write out the two regridded arrays to a daily netcdf file using
>>> method 1, Netcdf4, with file compression level set to 4
>>>
>>> The problem: The script fails toward the end (e.g, around the 27th file
>>> of a month) with the following error message and begins to input nonsense
>>> Temperature and radiation values into the array and states the weights file
>>> cannot be opened (it does exist). When I re-run the failed time-period as a
>>> one-time regridding, it regrids just fine. Is ncl unhappy because of the
>>> high I/O? It doesn't fail at the netcdf generation step ... it seems to
>>> fail at the regridding step itself. Thanks for any help! The error output
>>> follows ...
>>>
>>> Variable: hh
>>> Type: integer
>>> Total Size: 4 bytes
>>>             1 values
>>> Number of Dimensions: 1
>>> Dimensions and sizes:   [1]
>>> Coordinates:
>>> (0)     15
>>> fatal:["NclNewHDF5.c":3787]:NclNewHDF5.c: Cannot open file:
>>> </n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc>
>>> fatal:["NclAdvancedFile.c":2667]:_NclFileFillHLFS: Could not open
>>> (/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc)
>>> (0)   ESMF_regrid_with_weights: cannot open weight file
>>> '/n/regal/wofsy_lab/adayalu/T2SWDOWN_to_MODIS_bilinear.faster.nc'.
>>> (0)
>>> (0)     TEMP at 2 M: min=239.284   max=299.333
>>> (0)
>>> (0)     min=9.96921e+36   max=9.96921e+36
>>> warning:VarVarWrite: rhs has no dimension name or coordinate variable,
>>> deleting name of lhs dimension number (0) and destroying coordinate var,
>>>  use "(/../)" if this is not desired outcome
>>> warning:VarVarWrite: rhs has no dimension name or coordinate variable,
>>> deleting name of lhs dimension number (1) and destroying coordinate var,
>>>  use "(/../)" if this is not desired outcome
>>> warning:["Execute.c":8578]:Execute: Error occurred at or near line 64 in
>>> file regrid_using_weights_T2SWDOWN_MOD_noplots.ncl
>>>
>>> HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
>>>   #000: H5D.c line 461 in H5Dget_space(): not a dataset
>>>     major: Invalid arguments to routine
>>>     minor: Inappropriate type
>>> (...then a series of similar error messages are output .... and then it
>>> continues .... and nonsense is output):
>>>
>>> Variable: hh
>>> Type: integer
>>> Total Size: 4 bytes
>>>             1 values
>>> Number of Dimensions: 1
>>> Dimensions and sizes:   [1]
>>> Coordinates:
>>> (0)     17
>>> (0)
>>> (0)     TEMP at 2 M: min=0.0114805   max=299.333
>>> (0)
>>> (0)     DOWNWARD SHORT WAVE FLUX AT GROUND SURFACE: min=0
>>> max=2.90338e-31
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>
>
>
> --
> ____________________________________________________
> *Archana Dayalu*
> Graduate Student
> Dept. of Earth and Planetary Sciences
> Harvard University
> 24 Oxford Street #402
> Cambridge, MA 02138
> (617) 384-8206
>

-- 
____________________________________________________
*Archana Dayalu*
Graduate Student
Dept. of Earth and Planetary Sciences
Harvard University
24 Oxford Street #402
Cambridge, MA 02138
(617) 384-8206
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160112/7bff206c/attachment.html 