[ncl-talk] How to combine many nc files from different folders

[Dennis Shea]

Dave gave some good advice.

--
For fun: Test with varying numbers of files, For example

Change:
    fils= *systemfunc*
<https://www.ncl.ucar.edu/Document/Functions/Built-in/systemfunc.shtml>("ls
[1-2][8-9-0-1]*/*nc")
To:
   fils= *systemfunc*
<https://www.ncl.ucar.edu/Document/Functions/Built-in/systemfunc.shtml>("ls
198*/*nc")   ; 80s only
Then
   fils= *systemfunc*
<https://www.ncl.ucar.edu/Document/Functions/Built-in/systemfunc.shtml>("ls
199*/*nc")    ; 90s only

etc



print(fils)

On Fri, Sep 20, 2019 at 10:00 AM Setareh Rahimi via ncl-talk <
ncl-talk at ucar.edu> wrote:

> Dear Dave,
> Thank you for your suggestion. I think the problem comes from where  I
> combine so many NetCDF files together. However, I could not remove the
> warnings.
> Best wishes,
>
> On Thu, Sep 19, 2019 at 11:35 PM Dave Allured - NOAA Affiliate <
> dave.allured at noaa.gov> wrote:
>
>> That is good information from the diagnostics.  All file structure
>> details look fine to me.  However I notice that you are using addfiles to
>> open about 12780 files simultaneously.  I missed this the first time.  You
>> may be running into a system limit of number of simultaneous open files.
>>
>> This is discussed in a paragraph about half way down the documentation
>> page for the addfiles function.  From that, I suggest adding this line
>> before your addfiles command:
>>
>>      setfileoption ("nc", "SuppressClose", False)
>>
>> I am not familiar with this option.  It is possible that you need to
>> place this command after addfiles, and change the first argument from "nc"
>> to just "f" without quotes.  Try it both ways if necessary.
>>
>>
>> On Thu, Sep 19, 2019 at 10:35 AM Setareh Rahimi <setareh.rahimi at gmail.com>
>> wrote:
>>
>>> Dear all,
>>> Thank you for your suggestions. However, the problem still exists. I
>>> checked the files individually and found out nothing wrong with them. I run
>>> the script for each year separately and did not get the warnings, but once
>>> run the script for all the years together the warnings appear. I attached
>>> the output from tests that Dave has suggested.
>>> NCL version: 6.6.2
>>> Computer system: macOS Mojave, version 10.14
>>> Best wishes,
>>>
>>>
>>> On Wed, Sep 18, 2019 at 3:16 AM Dave Allured - NOAA Affiliate <
>>> dave.allured at noaa.gov> wrote:
>>>
>>>> Recently there are some known conditions that can cause unknown format
>>>> and corrupted file messages for valid netcdf files.  I recommend diagnosing
>>>> individual files, not using NCL, before dismissing an entire file set as
>>>> corrupted.  Try this black magic first and see if NCL can then read the
>>>> files:
>>>>
>>>>     Bash:       HDF5_USE_FILE_LOCKING=FALSE
>>>>     C-shell:    setenv HDF5_USE_FILE_LOCKING FALSE
>>>>
>>>> If that does not work, then try these tests.  The first two are
>>>> guaranteed to work on all file types.
>>>>
>>>>     file data.nc
>>>>     od -c -N16 data.nc
>>>>     h5dump -BH data.nc
>>>>     ncdump -k data.nc
>>>>     ncdump -sh data.nc
>>>>
>>>> If the problem has not become obvious, then post output from these
>>>> tests to this mailing list.  If more than 40 lines long, put all output
>>>> into a text file with name ending in .txt, and send as a file attachment to
>>>> your message.  Please do not send any screen shots.
>>>>
>>>> Also send your NCL version number and type of computer system.
>>>>
>>>> --Dave
>>>>
>>>>
>>>> On Tue, Sep 17, 2019 at 3:27 PM Dennis Shea via ncl-talk <
>>>> ncl-talk at ucar.edu> wrote:
>>>>
>>>>> As the message states, the file is "*corrupted*".  This is not an NCL
>>>>> issue.
>>>>>
>>>>> [1] delete the file[s] and try reacquiring them
>>>>> [2] Possibly, the source files are corrupted.
>>>>> ---
>>>>> FYI: There are some Persiann examples.
>>>>> *https://www.ncl.ucar.edu/Applications/HiResPrc.shtml*
>>>>> <https://www.ncl.ucar.edu/Applications/HiResPrc.shtml>
>>>>>
>>>>>
>>>>> On Tue, Sep 17, 2019 at 12:15 PM Setareh Rahimi <
>>>>> setareh.rahimi at gmail.com> wrote:
>>>>>
>>>>>> Dear Adam, and Dennis,
>>>>>> Thanks for your help. hopefully I could combine those files together,
>>>>>> but NCL tells there is something wrong with some files (attached image). In
>>>>>> order to check what could be wrong, I redownload 1983 files again and got
>>>>>> many warning for the second time. Any suggestion to remove those warning,
>>>>>> please?
>>>>>> Best wishes,
>>>>>>
>>>>>
>
> --
> S.Rahimi
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190920/a16b8602/attachment.html>