[ncl-talk] Fwd: Deal with a lot of netcdf files

[Dennis Shea]

re: 24 X 365 X 5 (24 files per day over 5 years)
===============
That is 43800 files.

Did you look at the FAQ???

http://www.ncl.ucar.edu/FAQ/

See the "File input/output" section
Question 26

===============
files   = systemfunc("ls wrf_1h/wrfout_d02_*") + ".nc"
nfiles = dimsizes(files)
print("nfiles="+nfiles)


setfileoption("nc", "SuppressClose", False)  ; READ documentation

a  = addfiles (files, "r")

===============
Do you *need* all 43800 files at one time?
You may have to rethink your approach.




On Thu, Jun 25, 2015 at 1:50 AM, Rabah Hachelaf <hachelaf at sca.uqam.ca>
wrote:

> 24 X 365 X 5 (24 files per day over 5 years)
>
>
> 2015-06-25 3:42 GMT-04:00 Lam Hoang <lamhpvn at gmail.com>:
>
>> How man files do you have?
>>
>>
>> On Thursday, June 25, 2015, Rabah Hachelaf <hachelaf at sca.uqam.ca> wrote:
>>
>>> ulimit -n 512 does't not solve the problem
>>>
>>> Any other advises ??
>>>
>>> 2015-06-24 23:47 GMT-04:00 Lam Hoang <lamhpvn at gmail.com>:
>>>
>>>> You should check the open files limit by using
>>>> ulimit -a
>>>>
>>>> You will find parameter: open files (-n) 256 with 256 is maximum open
>>>> files.
>>>> You can change this parameter with command:
>>>> ulimit -n 512
>>>>
>>>> Hope this help.
>>>> Lam
>>>>
>>>> On Thu, Jun 25, 2015 at 1:05 PM, Rabah Hachelaf <hachelaf at sca.uqam.ca>
>>>> wrote:
>>>>
>>>>>
>>>>> Hello NCL users,
>>>>>
>>>>> I have netcdf wrf outputs every 1h for 5 years,
>>>>> i use those commands to read files :
>>>>>
>>>>> files = systemfunc("ls wrf_1h/wrfout_d02_*") + ".nc"
>>>>> a  = addfiles (files, "r")
>>>>> ListSetType (a, "cat ")
>>>>>
>>>>> but i get a Segmentation fault error,
>>>>> when reduce the number of read files my script works correctly.
>>>>>
>>>>> Any suggestion about this issue.
>>>>>
>>>>> Thank you
>>>>> --
>>>>> ------------------------------
>>>>> Best regards,
>>>>> Rabah Hachelaf
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lam Hoang
>>>> Viet Nam National Center of Hydro-Meteorology Forecasting
>>>> National Hydro-Meteorological Services of Vietnam
>>>>
>>>> PhD Candidate
>>>> School of Earth, Atmosphere and Environment
>>>> Monash University, VIC 3800
>>>> Mob: +84 12 1616 5258 (VN)
>>>>          +61 4 5251 8668 (AUS)
>>>>
>>>>
>>>
>>>
>>> --
>>> ------------------------------
>>> Cordialement,
>>> Best regards,
>>> Rabah Hachelaf
>>>
>>
>>
>> --
>> Lam Hoang
>> Viet Nam National Center of Hydro-Meteorology Forecasting
>> National Hydro-Meteorological Services of Vietnam
>>
>> PhD Candidate
>> School of Earth, Atmosphere and Environment
>> Monash University, VIC 3800
>> Mob: +84 12 1616 5258 (VN)
>>          +61 4 5251 8668 (AUS)
>>
>>
>>
>
>
> --
> ------------------------------
> Cordialement,
> Best regards,
> Rabah Hachelaf
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150625/715229da/attachment.html