[ncl-talk] low memory algorithm for sorting ? how large is INT_MAX ?

[Adam Herrington]

the length of the array is:

npts = 5109033600

Yes, the problem is the array is too big. I am fairly certain that the
problem is npts > INT_MAX, where INT_MAX appears to be some limiter on
array size used in many of the ncl functions (I get this error all the
time).

My goal is to compute a percentile. I was thinking that by generating
INT_MAX samples of the array, and putting those samples into an array of
length INT_MAX, I can sort the reduced array and find the percentiles of
the reduced array. However, I would like to know what the value of INT_MAX
is. Do you know how I can find that out ?

Adam

On Fri, Oct 20, 2017 at 5:34 PM, Kevin Hallock <hallock at ucar.edu> wrote:

> Hi Adam,
>
> Thanks for sharing the error and the code snippet.
>
> Based on the error message, it seems like there may be an issue with the
> size of dtcond1d.
>
> Could you please add “print(npts)” before “ip = dim_pqsort(dtcond1d,kflag)”
> and rerun the script? I’m curious what NCL thinks the dimensionality of
> dtcond1d is.
>
> Thanks,
> Kevin
>
> On Oct 20, 2017, at 5:01 PM, Adam Herrington <adam.herrington at stonybrook.
> edu> wrote:
>
> the error message:
>
> fatal:dim_pqsort: one or more input dimensions sizes are greater than
> INT_MAX
> ^Mfatal:["Execute.c":8640]:Execute: Error occurred at or near line 294 in
> file omega3d_pdf_samples.ncl
>
> A snippet of the code:
>
>      dtcond1d = ndtooned(DTCOND)
>      delete(DTCOND)
>      npts = dimsizes(dtcond1d)
>      kflag=2
>      ip = dim_pqsort(dtcond1d,kflag)
>
>      ;;95th
>      tmp = npts*0.95
>      iptf = tointeger(floor(tmp))
>      dtlimhi(2) = dtcond1d(iptf)
>
>
>
>
> On Fri, Oct 20, 2017 at 4:17 PM, Kevin Hallock <hallock at ucar.edu> wrote:
>
>> Hi Adam,
>>
>> Could you please copy and paste the full error message you’re seeing?
>>
>> Also, would you mind sharing a copy of your script? It might help
>> diagnose the issue.
>>
>> Thanks,
>> Kevin
>>
>> > On Oct 20, 2017, at 3:49 PM, Adam Herrington <
>> adam.herrington at stonybrook.edu> wrote:
>> >
>> > I am trying to sort with the function 'dim_pqsort', but my array is too
>> large. I get an error saying that 1 or more dimensions is larger than
>> INT_MAX.
>> >
>> > If anyone has a solution to this problem, I'm all ears.
>> >
>> > My thinking is that I can create an array of sample indices through
>> 'generate_sample_indices', and make the number of sample indices = INT_MAX.
>> Does anyone know how large INT_MAX is ?
>> >
>> > Adam Herrington
>> > _______________________________________________
>> > ncl-talk mailing list
>> > ncl-talk at ucar.edu
>> > List instructions, subscriber options, unsubscribe:
>> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171020/9bb57f29/attachment.html>