[ncl-talk] low memory algorithm for sorting ? how large is INT_MAX ?

[Kevin Hallock]

Hi Adam,

As Gus mentioned, the value of INT_MAX is defined as 2147483647 (2^31 - 1) in the C Standard Library. Theoretically, NCL could be modified to support larger dimension sizes, but I don’t think it is currently possible to have dimensions larger than INT_MAX.

I will consult with the other NCL developers to see if this is something we might be able to add support for in the future.

Kevin

> On Oct 20, 2017, at 5:44 PM, Adam Herrington <adam.herrington at stonybrook.edu> wrote:
> 
> the length of the array is:
> 
> npts = 5109033600
> 
> Yes, the problem is the array is too big. I am fairly certain that the problem is npts > INT_MAX, where INT_MAX appears to be some limiter on array size used in many of the ncl functions (I get this error all the time).
> 
> My goal is to compute a percentile. I was thinking that by generating INT_MAX samples of the array, and putting those samples into an array of length INT_MAX, I can sort the reduced array and find the percentiles of the reduced array. However, I would like to know what the value of INT_MAX is. Do you know how I can find that out ?
> 
> Adam
> 
> On Fri, Oct 20, 2017 at 5:34 PM, Kevin Hallock <hallock at ucar.edu <mailto:hallock at ucar.edu>> wrote:
> Hi Adam,
> 
> Thanks for sharing the error and the code snippet.
> 
> Based on the error message, it seems like there may be an issue with the size of dtcond1d.
> 
> Could you please add “print(npts)” before “ip = dim_pqsort(dtcond1d,kflag)” and rerun the script? I’m curious what NCL thinks the dimensionality of dtcond1d is.
> 
> Thanks,
> Kevin
> 
>> On Oct 20, 2017, at 5:01 PM, Adam Herrington <adam.herrington at stonybrook.edu <mailto:adam.herrington at stonybrook.edu>> wrote:
>> 
>> the error message:
>> 
>> fatal:dim_pqsort: one or more input dimensions sizes are greater than INT_MAX
>> ^Mfatal:["Execute.c":8640]:Execute: Error occurred at or near line 294 in file omega3d_pdf_samples.ncl
>> 
>> A snippet of the code:
>> 
>>      dtcond1d = ndtooned(DTCOND)
>>      delete(DTCOND)
>>      npts = dimsizes(dtcond1d)
>>      kflag=2
>>      ip = dim_pqsort(dtcond1d,kflag)
>> 
>>      ;;95th
>>      tmp = npts*0.95
>>      iptf = tointeger(floor(tmp))
>>      dtlimhi(2) = dtcond1d(iptf)
>> 
>> 
>> 
>> 
>> On Fri, Oct 20, 2017 at 4:17 PM, Kevin Hallock <hallock at ucar.edu <mailto:hallock at ucar.edu>> wrote:
>> Hi Adam,
>> 
>> Could you please copy and paste the full error message you’re seeing?
>> 
>> Also, would you mind sharing a copy of your script? It might help diagnose the issue.
>> 
>> Thanks,
>> Kevin
>> 
>> > On Oct 20, 2017, at 3:49 PM, Adam Herrington <adam.herrington at stonybrook.edu <mailto:adam.herrington at stonybrook.edu>> wrote:
>> >
>> > I am trying to sort with the function 'dim_pqsort', but my array is too large. I get an error saying that 1 or more dimensions is larger than INT_MAX.
>> >
>> > If anyone has a solution to this problem, I'm all ears.
>> >
>> > My thinking is that I can create an array of sample indices through 'generate_sample_indices', and make the number of sample indices = INT_MAX. Does anyone know how large INT_MAX is ?
>> >
>> > Adam Herrington
>> > _______________________________________________
>> > ncl-talk mailing list
>> > ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
>> > List instructions, subscriber options, unsubscribe:
>> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171020/250a5e3c/attachment.html>