[ncl-talk] How to combine many nc files from different folders

[Dave Allured - NOAA Affiliate]

That is good information from the diagnostics.  All file structure details
look fine to me.  However I notice that you are using addfiles to open
about 12780 files simultaneously.  I missed this the first time.  You may
be running into a system limit of number of simultaneous open files.

This is discussed in a paragraph about half way down the documentation page
for the addfiles function.  From that, I suggest adding this line before
your addfiles command:

     setfileoption ("nc", "SuppressClose", False)

I am not familiar with this option.  It is possible that you need to place
this command after addfiles, and change the first argument from "nc" to
just "f" without quotes.  Try it both ways if necessary.


On Thu, Sep 19, 2019 at 10:35 AM Setareh Rahimi <setareh.rahimi at gmail.com>
wrote:

> Dear all,
> Thank you for your suggestions. However, the problem still exists. I
> checked the files individually and found out nothing wrong with them. I run
> the script for each year separately and did not get the warnings, but once
> run the script for all the years together the warnings appear. I attached
> the output from tests that Dave has suggested.
> NCL version: 6.6.2
> Computer system: macOS Mojave, version 10.14
> Best wishes,
>
>
> On Wed, Sep 18, 2019 at 3:16 AM Dave Allured - NOAA Affiliate <
> dave.allured at noaa.gov> wrote:
>
>> Recently there are some known conditions that can cause unknown format
>> and corrupted file messages for valid netcdf files.  I recommend diagnosing
>> individual files, not using NCL, before dismissing an entire file set as
>> corrupted.  Try this black magic first and see if NCL can then read the
>> files:
>>
>>     Bash:       HDF5_USE_FILE_LOCKING=FALSE
>>     C-shell:    setenv HDF5_USE_FILE_LOCKING FALSE
>>
>> If that does not work, then try these tests.  The first two are
>> guaranteed to work on all file types.
>>
>>     file data.nc
>>     od -c -N16 data.nc
>>     h5dump -BH data.nc
>>     ncdump -k data.nc
>>     ncdump -sh data.nc
>>
>> If the problem has not become obvious, then post output from these tests
>> to this mailing list.  If more than 40 lines long, put all output into a
>> text file with name ending in .txt, and send as a file attachment to your
>> message.  Please do not send any screen shots.
>>
>> Also send your NCL version number and type of computer system.
>>
>> --Dave
>>
>>
>> On Tue, Sep 17, 2019 at 3:27 PM Dennis Shea via ncl-talk <
>> ncl-talk at ucar.edu> wrote:
>>
>>> As the message states, the file is "*corrupted*".  This is not an NCL
>>> issue.
>>>
>>> [1] delete the file[s] and try reacquiring them
>>> [2] Possibly, the source files are corrupted.
>>> ---
>>> FYI: There are some Persiann examples.
>>> *https://www.ncl.ucar.edu/Applications/HiResPrc.shtml*
>>> <https://www.ncl.ucar.edu/Applications/HiResPrc.shtml>
>>>
>>>
>>> On Tue, Sep 17, 2019 at 12:15 PM Setareh Rahimi <
>>> setareh.rahimi at gmail.com> wrote:
>>>
>>>> Dear Adam, and Dennis,
>>>> Thanks for your help. hopefully I could combine those files together,
>>>> but NCL tells there is something wrong with some files (attached image). In
>>>> order to check what could be wrong, I redownload 1983 files again and got
>>>> many warning for the second time. Any suggestion to remove those warning,
>>>> please?
>>>> Best wishes,
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190919/2483ebbd/attachment.html>