[ncl-talk] How to combine many nc files from different folders

[Setareh Rahimi]

Dear Dave,
Thank you for your suggestion. I think the problem comes from where  I
combine so many NetCDF files together. However, I could not remove the
warnings.
Best wishes,

On Thu, Sep 19, 2019 at 11:35 PM Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> That is good information from the diagnostics.  All file structure details
> look fine to me.  However I notice that you are using addfiles to open
> about 12780 files simultaneously.  I missed this the first time.  You may
> be running into a system limit of number of simultaneous open files.
>
> This is discussed in a paragraph about half way down the documentation
> page for the addfiles function.  From that, I suggest adding this line
> before your addfiles command:
>
>      setfileoption ("nc", "SuppressClose", False)
>
> I am not familiar with this option.  It is possible that you need to place
> this command after addfiles, and change the first argument from "nc" to
> just "f" without quotes.  Try it both ways if necessary.
>
>
> On Thu, Sep 19, 2019 at 10:35 AM Setareh Rahimi <setareh.rahimi at gmail.com>
> wrote:
>
>> Dear all,
>> Thank you for your suggestions. However, the problem still exists. I
>> checked the files individually and found out nothing wrong with them. I run
>> the script for each year separately and did not get the warnings, but once
>> run the script for all the years together the warnings appear. I attached
>> the output from tests that Dave has suggested.
>> NCL version: 6.6.2
>> Computer system: macOS Mojave, version 10.14
>> Best wishes,
>>
>>
>> On Wed, Sep 18, 2019 at 3:16 AM Dave Allured - NOAA Affiliate <
>> dave.allured at noaa.gov> wrote:
>>
>>> Recently there are some known conditions that can cause unknown format
>>> and corrupted file messages for valid netcdf files.  I recommend diagnosing
>>> individual files, not using NCL, before dismissing an entire file set as
>>> corrupted.  Try this black magic first and see if NCL can then read the
>>> files:
>>>
>>>     Bash:       HDF5_USE_FILE_LOCKING=FALSE
>>>     C-shell:    setenv HDF5_USE_FILE_LOCKING FALSE
>>>
>>> If that does not work, then try these tests.  The first two are
>>> guaranteed to work on all file types.
>>>
>>>     file data.nc
>>>     od -c -N16 data.nc
>>>     h5dump -BH data.nc
>>>     ncdump -k data.nc
>>>     ncdump -sh data.nc
>>>
>>> If the problem has not become obvious, then post output from these tests
>>> to this mailing list.  If more than 40 lines long, put all output into a
>>> text file with name ending in .txt, and send as a file attachment to your
>>> message.  Please do not send any screen shots.
>>>
>>> Also send your NCL version number and type of computer system.
>>>
>>> --Dave
>>>
>>>
>>> On Tue, Sep 17, 2019 at 3:27 PM Dennis Shea via ncl-talk <
>>> ncl-talk at ucar.edu> wrote:
>>>
>>>> As the message states, the file is "*corrupted*".  This is not an NCL
>>>> issue.
>>>>
>>>> [1] delete the file[s] and try reacquiring them
>>>> [2] Possibly, the source files are corrupted.
>>>> ---
>>>> FYI: There are some Persiann examples.
>>>> *https://www.ncl.ucar.edu/Applications/HiResPrc.shtml*
>>>> <https://www.ncl.ucar.edu/Applications/HiResPrc.shtml>
>>>>
>>>>
>>>> On Tue, Sep 17, 2019 at 12:15 PM Setareh Rahimi <
>>>> setareh.rahimi at gmail.com> wrote:
>>>>
>>>>> Dear Adam, and Dennis,
>>>>> Thanks for your help. hopefully I could combine those files together,
>>>>> but NCL tells there is something wrong with some files (attached image). In
>>>>> order to check what could be wrong, I redownload 1983 files again and got
>>>>> many warning for the second time. Any suggestion to remove those warning,
>>>>> please?
>>>>> Best wishes,
>>>>>
>>>>

-- 
S.Rahimi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190920/221f1de3/attachment.html>