[ncl-talk] How to combine many nc files from different folders
Dave Allured - NOAA Affiliate
dave.allured at noaa.gov
Fri Oct 11 07:52:34 MDT 2019
(1) It looks like your current script can not find any of your files. Did
you customize the wildcard expression in line 1 to find the files on your
own computer? This is your responsibility.
(2) You did not send the complete output, and important messages were cut
off. Next time, please do not send a screen shot. I find them difficult
to work with. Send the complete output in a text file attachment, ending
in .txt filename suffix.
In general, I recommend studying each line of the example that I sent,
until you understand how each line works and what to expect for its
result. Use printVarSummary and print as needed, to help understand each
line. Good luck.
On Fri, Oct 11, 2019 at 1:18 AM Setareh Rahimi <setareh.rahimi at gmail.com>
wrote:
> Dear Dave,
> Thanks for your help and sorry for such a slow reply (too much busy). I
> run the code you have suggested, but getting an error. please have a look
> at the attached file.
> Best wishes,
>
> On Mon, Sep 23, 2019 at 11:57 PM Dave Allured - NOAA Affiliate <
> dave.allured at noaa.gov> wrote:
>
>> What do you mean by combine each year together? If you can condense each
>> year by computing intermediate results for each year in smaller arrays,
>> then yes this is an excellent approach to a large file problem. If you
>> mean read all daily grids in chunks, this is more difficult.
>>
>> The code sample that I sent before should work as well as reading all
>> daily grids in chunks. It will give helpful information about this
>> problem, whether it works or fails. Can you please let us know what
>> problem it generates, if any? Running too slow is a valid complaint in
>> this case.
>>
>> Please make two changes before running this again:
>>
>> (1) There is a mistake on line 5 below. If you did not already fix
>> this, please change to:
>>
>> dims3(0) = nfiles
>>
>> (2) For better diagnostics, change the print statement inside the loop
>> to this:
>>
>> print (systemfunc ("date") + " " + i)
>>
>>
>> On Mon, Sep 23, 2019 at 8:57 AM Setareh Rahimi <setareh.rahimi at gmail.com>
>> wrote:
>>
>>> Dear all,
>>>
>>> Is this possible to combine each year together separately, and then
>>> combine all of those combined years? Does it make sense? Please kindly
>>> advice me in this regard.
>>> Best wishes,
>>>
>>>
>>> On Fri, Sep 20, 2019 at 8:44 PM Dave Allured - NOAA Affiliate <
>>> dave.allured at noaa.gov> wrote:
>>>
>>>> You are trying to create an array of about 35 Gbytes in memory. This
>>>> is large and may be straining some limit inside the I/O system. This may
>>>> be difficult to debug directly.
>>>>
>>>> Instead try an alternate strategy, not using addfiles. Pre-allocate
>>>> the large array in NCL. Then read files one at a time in a loop, and
>>>> insert them into the large array. Something like this (not tested):
>>>>
>>>> fils = systemfunc ("ls /*/*.nc")
>>>> f = addfile (fils(0), "r")
>>>> dims3 = getfilevardimsizes (f, "p")
>>>> nfiles = dimsizes (fils)
>>>> dims(0) = nfiles
>>>> print (dims3)
>>>>
>>>> x = new (dims3, "float")
>>>> printVarSummary (x)
>>>>
>>>> do i = 0, nfiles-1
>>>> print (i+"")
>>>> f = addfile (fils(i), "r")
>>>> x(i,:,:) = f->p
>>>> end do
>>>>
>>>> printVarSummary (x)
>>>> printMinMax (x,0)
>>>>
>>>> This will probably avoid memory problems in the I/O system, but you
>>>> might run into an out of memory problem in the NCL core. This test program
>>>> will help to localize the problem.
>>>>
>>>> It is also time to check more version information. How was NCL on your
>>>> system built and installed? Can you get the version numbers of the netcdf
>>>> and HDF5 libraries that NCL was built with? There were hints in previous
>>>> diagnostics that you may be using older library versions. Caution, the
>>>> library versions linked with NCL may not be the same versions used in
>>>> command line tools h5dump and ncdump.
>>>>
>>>> Also what is the physical memory size (RAM) in your Mac?
>>>>
>>>>
>>>> On Fri, Sep 20, 2019 at 8:00 AM Setareh Rahimi <
>>>> setareh.rahimi at gmail.com> wrote:
>>>>
>>>>> Dear Dave,
>>>>> Thank you for your suggestion. I think the problem comes from where I
>>>>> combine so many NetCDF files together. However, I could not remove the
>>>>> warnings.
>>>>> Best wishes,
>>>>>
>>>>> On Thu, Sep 19, 2019 at 11:35 PM Dave Allured - NOAA Affiliate <
>>>>> dave.allured at noaa.gov> wrote:
>>>>>
>>>>>> That is good information from the diagnostics. All file structure
>>>>>> details look fine to me. However I notice that you are using addfiles to
>>>>>> open about 12780 files simultaneously. I missed this the first time. You
>>>>>> may be running into a system limit of number of simultaneous open files.
>>>>>>
>>>>>> This is discussed in a paragraph about half way down the
>>>>>> documentation page for the addfiles function. From that, I suggest adding
>>>>>> this line before your addfiles command:
>>>>>>
>>>>>> setfileoption ("nc", "SuppressClose", False)
>>>>>>
>>>>>> I am not familiar with this option. It is possible that you need to
>>>>>> place this command after addfiles, and change the first argument from "nc"
>>>>>> to just "f" without quotes. Try it both ways if necessary.
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 19, 2019 at 10:35 AM Setareh Rahimi <
>>>>>> setareh.rahimi at gmail.com> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>> Thank you for your suggestions. However, the problem still exists. I
>>>>>>> checked the files individually and found out nothing wrong with them. I run
>>>>>>> the script for each year separately and did not get the warnings, but once
>>>>>>> run the script for all the years together the warnings appear. I attached
>>>>>>> the output from tests that Dave has suggested.
>>>>>>> NCL version: 6.6.2
>>>>>>> Computer system: macOS Mojave, version 10.14
>>>>>>> Best wishes,
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 18, 2019 at 3:16 AM Dave Allured - NOAA Affiliate <
>>>>>>> dave.allured at noaa.gov> wrote:
>>>>>>>
>>>>>>>> Recently there are some known conditions that can cause unknown
>>>>>>>> format and corrupted file messages for valid netcdf files. I recommend
>>>>>>>> diagnosing individual files, not using NCL, before dismissing an entire
>>>>>>>> file set as corrupted. Try this black magic first and see if NCL can then
>>>>>>>> read the files:
>>>>>>>>
>>>>>>>> Bash: HDF5_USE_FILE_LOCKING=FALSE
>>>>>>>> C-shell: setenv HDF5_USE_FILE_LOCKING FALSE
>>>>>>>>
>>>>>>>> If that does not work, then try these tests. The first two are
>>>>>>>> guaranteed to work on all file types.
>>>>>>>>
>>>>>>>> file data.nc
>>>>>>>> od -c -N16 data.nc
>>>>>>>> h5dump -BH data.nc
>>>>>>>> ncdump -k data.nc
>>>>>>>> ncdump -sh data.nc
>>>>>>>>
>>>>>>>> If the problem has not become obvious, then post output from these
>>>>>>>> tests to this mailing list. If more than 40 lines long, put all output
>>>>>>>> into a text file with name ending in .txt, and send as a file attachment to
>>>>>>>> your message. Please do not send any screen shots.
>>>>>>>>
>>>>>>>> Also send your NCL version number and type of computer system.
>>>>>>>>
>>>>>>>> --Dave
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 17, 2019 at 3:27 PM Dennis Shea via ncl-talk <
>>>>>>>> ncl-talk at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>> As the message states, the file is "*corrupted*". This is not an
>>>>>>>>> NCL issue.
>>>>>>>>>
>>>>>>>>> [1] delete the file[s] and try reacquiring them
>>>>>>>>> [2] Possibly, the source files are corrupted.
>>>>>>>>> ---
>>>>>>>>> FYI: There are some Persiann examples.
>>>>>>>>> *https://www.ncl.ucar.edu/Applications/HiResPrc.shtml*
>>>>>>>>> <https://www.ncl.ucar.edu/Applications/HiResPrc.shtml>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 17, 2019 at 12:15 PM Setareh Rahimi <
>>>>>>>>> setareh.rahimi at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Dear Adam, and Dennis,
>>>>>>>>>> Thanks for your help. hopefully I could combine those files
>>>>>>>>>> together, but NCL tells there is something wrong with some files (attached
>>>>>>>>>> image). In order to check what could be wrong, I redownload 1983 files
>>>>>>>>>> again and got many warning for the second time. Any suggestion to remove
>>>>>>>>>> those warning, please?
>>>>>>>>>> Best wishes,
>>>>>>>>>>
>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20191011/f9926c54/attachment.html>
More information about the ncl-talk
mailing list