<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Adam,<div class=""><br class=""></div><div class="">I am writing back to you to clarify what I said last week.</div><div class=""><br class=""></div><div class="">NCL <i class="">does</i> support dimension sizes larger than 2147483647, as evidenced by the “dtcond1d = ndtooned(DTCOND)” and “npts = dimsizes(dtcond1d)” lines working.</div><div class=""><br class=""></div><div class="">The real issue is that certain NCL computational functions (“dim_pqsort()” in this case) are hard coded to reject variables where “ndim > INT_MAX”.</div><div class=""><br class=""></div><div class="">Support for dimensions larger than INT_MAX already exists in the data structure that represents an NCL variable under the hood. The remaining work to be done is just finding older functions that were never updated to accommodate these larger dimensions. I can’t guarantee a timeframe for all of these functions to be updated, but I’ll make a note for dim_pqsort() to be given higher priority.</div><div class=""><br class=""></div><div class="">Kevin<br class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Oct 20, 2017, at 6:14 PM, Kevin Hallock <<a href="mailto:hallock@ucar.edu" class="">hallock@ucar.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Adam,<div class=""><br class=""></div><div class="">As Gus mentioned, the value of INT_MAX is defined as 2147483647 (2^31 - 1) in the C Standard Library. Theoretically, NCL could be modified to support larger dimension sizes, but I don’t think it is currently possible to have dimensions larger than INT_MAX.</div><div class=""><br class=""></div><div class="">I will consult with the other NCL developers to see if this is something we might be able to add support for in the future.</div><div class=""><br class=""></div><div class="">Kevin</div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Oct 20, 2017, at 5:44 PM, Adam Herrington <<a href="mailto:adam.herrington@stonybrook.edu" class="">adam.herrington@stonybrook.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">the length of the array is:<div class=""><br class=""></div><div class="">npts = 5109033600<br class=""></div><div class=""><br class=""></div><div class="">Yes, the problem is the array is too big. I am fairly certain that the problem is npts > INT_MAX, where INT_MAX appears to be some limiter on array size used in many of the ncl functions (I get this error all the time).</div><div class=""><br class=""></div><div class="">My goal is to compute a percentile. I was thinking that by generating INT_MAX samples of the array, and putting those samples into an array of length INT_MAX, I can sort the reduced array and find the percentiles of the reduced array. However, I would like to know what the value of INT_MAX is. Do you know how I can find that out ?</div><div class=""><br class=""></div><div class="">Adam</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Oct 20, 2017 at 5:34 PM, Kevin Hallock <span dir="ltr" class=""><<a href="mailto:hallock@ucar.edu" target="_blank" class="">hallock@ucar.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">Hi Adam,<div class=""><br class=""></div><div class="">Thanks for sharing the error and the code snippet.</div><div class=""><br class=""></div><div class="">Based on the error message, it seems like there may be an issue with the size of dtcond1d.</div><div class=""><br class=""></div><div class="">Could you please add “<font face="Menlo" class="">print(npts)</font>” before “<font face="Menlo" class="">ip = dim_pqsort(dtcond1d,kflag)</font>” and rerun the script? I’m curious what NCL thinks the dimensionality of dtcond1d is.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Kevin</div><div class=""><div class="h5"><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Oct 20, 2017, at 5:01 PM, Adam Herrington <<a href="mailto:adam.herrington@stonybrook.edu" target="_blank" class="">adam.herrington@stonybrook.<wbr class="">edu</a>> wrote:</div><br class="m_-6665440205293727416Apple-interchange-newline"><div class=""><div dir="ltr" class="">the error message:<div class=""><br class=""></div><div class=""><div class="">fatal:dim_pqsort: one or more input dimensions sizes are greater than INT_MAX</div><div class="">^Mfatal:["Execute.c":8640]:<wbr class="">Execute: Error occurred at or near line 294 in file omega3d_pdf_samples.ncl</div></div><div class=""><br class=""></div><div class="">A snippet of the code:</div><div class=""><br class=""></div><div class=""><div class=""> dtcond1d = ndtooned(DTCOND)</div><div class=""> delete(DTCOND)</div><div class=""> npts = dimsizes(dtcond1d)</div><div class=""> kflag=2</div><div class=""> ip = dim_pqsort(dtcond1d,kflag)</div><div class=""><br class=""></div><div class=""> ;;95th</div><div class=""> tmp = npts*0.95</div><div class=""> iptf = tointeger(floor(tmp))</div><div class=""> dtlimhi(2) = dtcond1d(iptf)</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Oct 20, 2017 at 4:17 PM, Kevin Hallock <span dir="ltr" class=""><<a href="mailto:hallock@ucar.edu" target="_blank" class="">hallock@ucar.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Adam,<br class="">
<br class="">
Could you please copy and paste the full error message you’re seeing?<br class="">
<br class="">
Also, would you mind sharing a copy of your script? It might help diagnose the issue.<br class="">
<br class="">
Thanks,<br class="">
Kevin<br class="">
<div class=""><div class="m_-6665440205293727416h5"><br class="">
> On Oct 20, 2017, at 3:49 PM, Adam Herrington <<a href="mailto:adam.herrington@stonybrook.edu" target="_blank" class="">adam.herrington@stonybrook.ed<wbr class="">u</a>> wrote:<br class="">
><br class="">
> I am trying to sort with the function 'dim_pqsort', but my array is too large. I get an error saying that 1 or more dimensions is larger than INT_MAX.<br class="">
><br class="">
> If anyone has a solution to this problem, I'm all ears.<br class="">
><br class="">
> My thinking is that I can create an array of sample indices through 'generate_sample_indices', and make the number of sample indices = INT_MAX. Does anyone know how large INT_MAX is ?<br class="">
><br class="">
> Adam Herrington<br class="">
</div></div>> ______________________________<wbr class="">_________________<br class="">
> ncl-talk mailing list<br class="">
> <a href="mailto:ncl-talk@ucar.edu" target="_blank" class="">ncl-talk@ucar.edu</a><br class="">
> List instructions, subscriber options, unsubscribe:<br class="">
> <a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank" class="">http://mailman.ucar.edu/mailma<wbr class="">n/listinfo/ncl-talk</a><br class="">
<br class="">
</blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></div></div></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></body></html>