[ncl-talk] How to combine many nc files from different folders

[Setareh Rahimi]

Dear Dave,

Thank you. I do not use NCL on mac, I used it on a server. I think there is
enough memory on the server.
Best wishes,

On Fri, Sep 20, 2019 at 8:44 PM Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> You are trying to create an array of about 35 Gbytes in memory.  This is
> large and may be straining some limit inside the I/O system.  This may be
> difficult to debug directly.
>
> Instead try an alternate strategy, not using addfiles.  Pre-allocate the
> large array in NCL.  Then read files one at a time in a loop, and insert
> them into the large array.  Something like this (not tested):
>
>    fils = systemfunc ("ls /*/*.nc")
>    f = addfile (fils(0), "r")
>    dims3 = getfilevardimsizes (f, "p")
>    nfiles = dimsizes (fils)
>    dims(0) = nfiles
>    print (dims3)
>
>    x = new (dims3, "float")
>    printVarSummary (x)
>
>    do i = 0, nfiles-1
>       print (i+"")
>       f = addfile (fils(i), "r")
>       x(i,:,:) = f->p
>    end do
>
>    printVarSummary (x)
>    printMinMax (x,0)
>
> This will probably avoid memory problems in the I/O system, but you might
> run into an out of memory problem in the NCL core.  This test program will
> help to localize the problem.
>
> It is also time to check more version information.  How was NCL on your
> system built and installed?  Can you get the version numbers of the netcdf
> and HDF5 libraries that NCL was built with?  There were hints in previous
> diagnostics that you may be using older library versions.  Caution, the
> library versions linked with NCL may not be the same versions used in
> command line tools h5dump and ncdump.
>
> Also what is the physical memory size (RAM) in your Mac?
>
>
> On Fri, Sep 20, 2019 at 8:00 AM Setareh Rahimi <setareh.rahimi at gmail.com>
> wrote:
>
>> Dear Dave,
>> Thank you for your suggestion. I think the problem comes from where  I
>> combine so many NetCDF files together. However, I could not remove the
>> warnings.
>> Best wishes,
>>
>> On Thu, Sep 19, 2019 at 11:35 PM Dave Allured - NOAA Affiliate <
>> dave.allured at noaa.gov> wrote:
>>
>>> That is good information from the diagnostics.  All file structure
>>> details look fine to me.  However I notice that you are using addfiles to
>>> open about 12780 files simultaneously.  I missed this the first time.  You
>>> may be running into a system limit of number of simultaneous open files.
>>>
>>> This is discussed in a paragraph about half way down the documentation
>>> page for the addfiles function.  From that, I suggest adding this line
>>> before your addfiles command:
>>>
>>>      setfileoption ("nc", "SuppressClose", False)
>>>
>>> I am not familiar with this option.  It is possible that you need to
>>> place this command after addfiles, and change the first argument from "nc"
>>> to just "f" without quotes.  Try it both ways if necessary.
>>>
>>>
>>> On Thu, Sep 19, 2019 at 10:35 AM Setareh Rahimi <
>>> setareh.rahimi at gmail.com> wrote:
>>>
>>>> Dear all,
>>>> Thank you for your suggestions. However, the problem still exists. I
>>>> checked the files individually and found out nothing wrong with them. I run
>>>> the script for each year separately and did not get the warnings, but once
>>>> run the script for all the years together the warnings appear. I attached
>>>> the output from tests that Dave has suggested.
>>>> NCL version: 6.6.2
>>>> Computer system: macOS Mojave, version 10.14
>>>> Best wishes,
>>>>
>>>>
>>>> On Wed, Sep 18, 2019 at 3:16 AM Dave Allured - NOAA Affiliate <
>>>> dave.allured at noaa.gov> wrote:
>>>>
>>>>> Recently there are some known conditions that can cause unknown format
>>>>> and corrupted file messages for valid netcdf files.  I recommend diagnosing
>>>>> individual files, not using NCL, before dismissing an entire file set as
>>>>> corrupted.  Try this black magic first and see if NCL can then read the
>>>>> files:
>>>>>
>>>>>     Bash:       HDF5_USE_FILE_LOCKING=FALSE
>>>>>     C-shell:    setenv HDF5_USE_FILE_LOCKING FALSE
>>>>>
>>>>> If that does not work, then try these tests.  The first two are
>>>>> guaranteed to work on all file types.
>>>>>
>>>>>     file data.nc
>>>>>     od -c -N16 data.nc
>>>>>     h5dump -BH data.nc
>>>>>     ncdump -k data.nc
>>>>>     ncdump -sh data.nc
>>>>>
>>>>> If the problem has not become obvious, then post output from these
>>>>> tests to this mailing list.  If more than 40 lines long, put all output
>>>>> into a text file with name ending in .txt, and send as a file attachment to
>>>>> your message.  Please do not send any screen shots.
>>>>>
>>>>> Also send your NCL version number and type of computer system.
>>>>>
>>>>> --Dave
>>>>>
>>>>>
>>>>> On Tue, Sep 17, 2019 at 3:27 PM Dennis Shea via ncl-talk <
>>>>> ncl-talk at ucar.edu> wrote:
>>>>>
>>>>>> As the message states, the file is "*corrupted*".  This is not an
>>>>>> NCL issue.
>>>>>>
>>>>>> [1] delete the file[s] and try reacquiring them
>>>>>> [2] Possibly, the source files are corrupted.
>>>>>> ---
>>>>>> FYI: There are some Persiann examples.
>>>>>> *https://www.ncl.ucar.edu/Applications/HiResPrc.shtml*
>>>>>> <https://www.ncl.ucar.edu/Applications/HiResPrc.shtml>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 17, 2019 at 12:15 PM Setareh Rahimi <
>>>>>> setareh.rahimi at gmail.com> wrote:
>>>>>>
>>>>>>> Dear Adam, and Dennis,
>>>>>>> Thanks for your help. hopefully I could combine those files
>>>>>>> together, but NCL tells there is something wrong with some files (attached
>>>>>>> image). In order to check what could be wrong, I redownload 1983 files
>>>>>>> again and got many warning for the second time. Any suggestion to remove
>>>>>>> those warning, please?
>>>>>>> Best wishes,
>>>>>>>
>>>>>>
>>
>> --
>> S.Rahimi
>>
>>

-- 
S.Rahimi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190923/2a96712e/attachment.html>