[ncl-talk] low memory algorithm for sorting ? how large is INT_MAX ?

[Kevin Hallock]

Hi Adam,

I am writing back to you to clarify what I said last week.

NCL does support dimension sizes larger than 2147483647, as evidenced by the “dtcond1d = ndtooned(DTCOND)” and “npts = dimsizes(dtcond1d)” lines working.

The real issue is that certain NCL computational functions (“dim_pqsort()” in this case) are hard coded to reject variables where “ndim > INT_MAX”.

Support for dimensions larger than INT_MAX already exists in the data structure that represents an NCL variable under the hood. The remaining work to be done is just finding older functions that were never updated to accommodate these larger dimensions. I can’t guarantee a timeframe for all of these functions to be updated, but I’ll make a note for dim_pqsort() to be given higher priority.

Kevin

> On Oct 20, 2017, at 6:14 PM, Kevin Hallock <hallock at ucar.edu> wrote:
> 
> Hi Adam,
> 
> As Gus mentioned, the value of INT_MAX is defined as 2147483647 (2^31 - 1) in the C Standard Library. Theoretically, NCL could be modified to support larger dimension sizes, but I don’t think it is currently possible to have dimensions larger than INT_MAX.
> 
> I will consult with the other NCL developers to see if this is something we might be able to add support for in the future.
> 
> Kevin
> 
>> On Oct 20, 2017, at 5:44 PM, Adam Herrington <adam.herrington at stonybrook.edu <mailto:adam.herrington at stonybrook.edu>> wrote:
>> 
>> the length of the array is:
>> 
>> npts = 5109033600
>> 
>> Yes, the problem is the array is too big. I am fairly certain that the problem is npts > INT_MAX, where INT_MAX appears to be some limiter on array size used in many of the ncl functions (I get this error all the time).
>> 
>> My goal is to compute a percentile. I was thinking that by generating INT_MAX samples of the array, and putting those samples into an array of length INT_MAX, I can sort the reduced array and find the percentiles of the reduced array. However, I would like to know what the value of INT_MAX is. Do you know how I can find that out ?
>> 
>> Adam
>> 
>> On Fri, Oct 20, 2017 at 5:34 PM, Kevin Hallock <hallock at ucar.edu <mailto:hallock at ucar.edu>> wrote:
>> Hi Adam,
>> 
>> Thanks for sharing the error and the code snippet.
>> 
>> Based on the error message, it seems like there may be an issue with the size of dtcond1d.
>> 
>> Could you please add “print(npts)” before “ip = dim_pqsort(dtcond1d,kflag)” and rerun the script? I’m curious what NCL thinks the dimensionality of dtcond1d is.
>> 
>> Thanks,
>> Kevin
>> 
>>> On Oct 20, 2017, at 5:01 PM, Adam Herrington <adam.herrington at stonybrook.edu <mailto:adam.herrington at stonybrook.edu>> wrote:
>>> 
>>> the error message:
>>> 
>>> fatal:dim_pqsort: one or more input dimensions sizes are greater than INT_MAX
>>> ^Mfatal:["Execute.c":8640]:Execute: Error occurred at or near line 294 in file omega3d_pdf_samples.ncl
>>> 
>>> A snippet of the code:
>>> 
>>>      dtcond1d = ndtooned(DTCOND)
>>>      delete(DTCOND)
>>>      npts = dimsizes(dtcond1d)
>>>      kflag=2
>>>      ip = dim_pqsort(dtcond1d,kflag)
>>> 
>>>      ;;95th
>>>      tmp = npts*0.95
>>>      iptf = tointeger(floor(tmp))
>>>      dtlimhi(2) = dtcond1d(iptf)
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Oct 20, 2017 at 4:17 PM, Kevin Hallock <hallock at ucar.edu <mailto:hallock at ucar.edu>> wrote:
>>> Hi Adam,
>>> 
>>> Could you please copy and paste the full error message you’re seeing?
>>> 
>>> Also, would you mind sharing a copy of your script? It might help diagnose the issue.
>>> 
>>> Thanks,
>>> Kevin
>>> 
>>> > On Oct 20, 2017, at 3:49 PM, Adam Herrington <adam.herrington at stonybrook.edu <mailto:adam.herrington at stonybrook.edu>> wrote:
>>> >
>>> > I am trying to sort with the function 'dim_pqsort', but my array is too large. I get an error saying that 1 or more dimensions is larger than INT_MAX.
>>> >
>>> > If anyone has a solution to this problem, I'm all ears.
>>> >
>>> > My thinking is that I can create an array of sample indices through 'generate_sample_indices', and make the number of sample indices = INT_MAX. Does anyone know how large INT_MAX is ?
>>> >
>>> > Adam Herrington
>>> > _______________________________________________
>>> > ncl-talk mailing list
>>> > ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
>>> > List instructions, subscriber options, unsubscribe:
>>> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>>> 
>>> 
>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171024/eb6aa195/attachment.html>