[ncl-talk] Joining large files using "cat" option

[Mary Haley]

Amit,

You are effectively trying to write a 16 GB file to the system.  It sounds
like your system is not going to allow this for whatever reason; maybe you
are running out of disk space.

One other thing to try is to add a setfileoption call:

   *setfileoption*("nc","Format","LargeFile")

You need to do this before you call "addfile".  I think I already mentioned
this in a previous message.

--Mary

On Tue, Dec 1, 2015 at 1:06 PM, amit timilsina <
timilsinaamit87 at huskers.unl.edu> wrote:

> Respected Mary,
>
>
> I am using 6.3.0 version of NCL in 64 bit Mac. Here, I am trying to
> concatenate at least 30 years files (like 2006 to 2035) or 100 years if
> possible (each file on yearly basis and has size 168.2 MB). I have attached
> the syntax I have used.
>
>
>
> Thank you
>
>
>
>
> *Amit Prasad Timilsina*
>
> PhD student
>
> School of Natural Resources,UNL
>
> 243 Hardin Hall
>
> 3310 Holdrege Street
>
> Lincoln, Nebraska
>
>
> ------------------------------
> *From:* Mary Haley <haley at ucar.edu>
> *Sent:* Tuesday, December 1, 2015 10:15 AM
> *To:* amit timilsina
> *Cc:* ncl-talk at ucar.edu
> *Subject:* Re: Joining large files using "cat" option
>
> Amit,
>
> [Sorry, I forgot to cc ncl-talk on the previous message.]
>
> We are a bit puzzled by the "File too large" error. We believe this is a
> UNIX error that is indicating your file is too large to be written on your
> system.
>
> Do you have some kind of quota limit on the size of files you can write?
> Or, are you by any chance running on a 32-bit system?
>
> What is the size of "pr"?  You can do a "printVarSummary" on pr to get
> this information.
>
> --Mary
>
>
> On Mon, Nov 30, 2015 at 12:27 PM, amit timilsina <
> timilsinaamit87 at huskers.unl.edu> wrote:
>
>> Respected Mary,
>>
>> Thank you very much. I tried to many files using "cat" option. I got
>> error like "File too large: error attempting to write variable (pr) to file
>> (pr_day_CMCC-CM_rcp45_r1i1p1_20060101-20351231.nc)". It seems that there is
>> a limitation on a number of files and files sizes. Is there any way to
>> solve this problem? Thank you very much for your time.
>>
>>
>> ncl 2> f=addfiles(fils,"r")
>>
>> ncl 3> ListSetType(f,"cat")
>>
>> ncl 4> pr=f[:]->pr
>>
>> ncl 5>
>> ncdf=addfile("pr_day_CMCC-CM_rcp45_r1i1p1_20060101-20351231.nc","c")
>>
>> ncl 6> filedimdef(ncdf,"time",-1,True)
>>
>> ncl 7> ncdf->pr=pr
>>
>> fatal:File too large: error attempting to write variable (pr) to file
>> (pr_day_CMCC-CM_rcp45_r1i1p1_20060101-20351231.nc)
>>
>> fatal:["Execute.c":8573]:Execute: Error occurred at or near line 7
>>
>>
>>
>>
>> Amit Prasad Timilsina
>> PhD student
>> School of Natural Resources,UNL
>> 243 Hardin Hall
>> 3310 Holdrege Street
>> Lincoln, Nebraska
>>
>>
>>
>> ------------------------------
>> *From:* Mary Haley <haley at ucar.edu>
>> *Sent:* Monday, November 30, 2015 11:22 AM
>> *To:* amit timilsina
>> *Cc:* ncl-talk at ucar.edu
>> *Subject:* Re: [ncl-talk] Error in joining files having
>>
>> Amit,
>>
>> The error is telling you that your coordinate variable is non-monotonic.
>> In order to use "cat" mode to join a dimension, which is usually the time
>> dimension, the time values must be monotonic. Also, I believe the time
>> arrays must have the same units.
>>
>> Please look at your "time" dimension on both files and make sure 1) they
>> are monotonic, 2) they have the same units, and 3) the first time value of
>> the second file is greater than the last time value of the first file.
>>
>> --Mary
>>
>>
>>
>>
>> On Fri, Nov 27, 2015 at 4:23 PM, amit timilsina <
>> timilsinaamit87 at huskers.unl.edu> wrote:
>>
>>> Respected all,
>>>
>>> Is the warning " warning:error attempting to fix non-monotonic
>>> aggregation variable"  while joining files makes any differences? The first
>>> file has dimension size: 366x240X480 and second file have 365x240x480 or
>>> first-year data is leap year data. Even with using both non-leap year data,
>>> I found the similar problem. What is the reason for getting this problem
>>> and how can I solve it?
>>>
>>> ncl 0> fils =
>>> (/"pr_day_CMCC-CM_decadal2005_r1i2p1_20120101-20121231.nc","pr_day_CMCC-CM_decadal2005_r1i2p1_20130101-20131231.nc"/)ncl
>>> 1> f=addfiles(fils,"r")
>>> ncl 2> ListSetType(f,"cat")
>>> ncl 3> pr=f[:]->pr
>>> warning:error attempting to fix non-monotonic aggregation variable
>>>
>>>
>>> Thank you in advance.
>>>
>>> Amit Prasad Timilsina
>>> PhD student
>>> School of Natural Resources,UNL
>>> 243 Hardin Hall
>>> 3310 Holdrege Street
>>> Lincoln, Nebraska
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20151201/ba8433d8/attachment.html