[ncl-talk] low memory algorithm for sorting ? how large is INT_MAX ?

[Gus Correa]

On 10/20/2017 07:44 PM, Adam Herrington wrote:
> the length of the array is:
> 
> npts = 5109033600
> 
> Yes, the problem is the array is too big. I am fairly certain that the 
> problem is npts > INT_MAX, where INT_MAX appears to be some limiter on 
> array size used in many of the ncl functions (I get this error all the 
> time).
> 
> My goal is to compute a percentile. I was thinking that by generating 
> INT_MAX samples of the array, and putting those samples into an array of 
> length INT_MAX, I can sort the reduced array and find the percentiles of 
> the reduced array. However, I would like to know what the value of 
> INT_MAX is. Do you know how I can find that out ?
> 

Hi

In Linux (at least current 64-bit versions) INT_MAX is 2^31-1 = 
2147483647, the maximum signed integer representable with 32 bits.
That is smaller than your array size.




> Adam
> 
> On Fri, Oct 20, 2017 at 5:34 PM, Kevin Hallock <hallock at ucar.edu 
> <mailto:hallock at ucar.edu>> wrote:
> 
>     Hi Adam,
> 
>     Thanks for sharing the error and the code snippet.
> 
>     Based on the error message, it seems like there may be an issue with
>     the size of dtcond1d.
> 
>     Could you please add “print(npts)” before “ip
>     = dim_pqsort(dtcond1d,kflag)” and rerun the script? I’m curious what
>     NCL thinks the dimensionality of dtcond1d is.
> 
>     Thanks,
>     Kevin
> 
>>     On Oct 20, 2017, at 5:01 PM, Adam Herrington
>>     <adam.herrington at stonybrook.edu
>>     <mailto:adam.herrington at stonybrook.edu>> wrote:
>>
>>     the error message:
>>
>>     fatal:dim_pqsort: one or more input dimensions sizes are greater
>>     than INT_MAX
>>     ^Mfatal:["Execute.c":8640]:Execute: Error occurred at or near line
>>     294 in file omega3d_pdf_samples.ncl
>>
>>     A snippet of the code:
>>
>>          dtcond1d = ndtooned(DTCOND)
>>          delete(DTCOND)
>>          npts = dimsizes(dtcond1d)
>>          kflag=2
>>          ip = dim_pqsort(dtcond1d,kflag)
>>
>>          ;;95th
>>          tmp = npts*0.95
>>          iptf = tointeger(floor(tmp))
>>          dtlimhi(2) = dtcond1d(iptf)
>>
>>
>>
>>
>>     On Fri, Oct 20, 2017 at 4:17 PM, Kevin Hallock <hallock at ucar.edu
>>     <mailto:hallock at ucar.edu>> wrote:
>>
>>         Hi Adam,
>>
>>         Could you please copy and paste the full error message you’re
>>         seeing?
>>
>>         Also, would you mind sharing a copy of your script? It might
>>         help diagnose the issue.
>>
>>         Thanks,
>>         Kevin
>>
>>         > On Oct 20, 2017, at 3:49 PM, Adam Herrington
>>         <adam.herrington at stonybrook.edu
>>         <mailto:adam.herrington at stonybrook.edu>> wrote:
>>         >
>>         > I am trying to sort with the function 'dim_pqsort', but my
>>         array is too large. I get an error saying that 1 or more
>>         dimensions is larger than INT_MAX.
>>         >
>>         > If anyone has a solution to this problem, I'm all ears.
>>         >
>>         > My thinking is that I can create an array of sample indices
>>         through 'generate_sample_indices', and make the number of
>>         sample indices = INT_MAX. Does anyone know how large INT_MAX is ?
>>         >
>>         > Adam Herrington
>>         > _______________________________________________
>>         > ncl-talk mailing list
>>         > ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
>>         > List instructions, subscriber options, unsubscribe:
>>         > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>         <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>>
>>
> 
> 
> 
> 
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>