[ncl-talk] Code Compiling Extremely Slowly
Zach Rieck
zrr817 at gmail.com
Fri Aug 9 12:15:33 MDT 2019
I'll take a look at this and see if it helps at all. As always, thank you
for your help Dennis!
-Zach Rieck
zrr817 at gmail.com
(513)-502-5652
On Wed, Aug 7, 2019 at 10:05 AM Dennis Shea <shea at ucar.edu> wrote:
> [1] If you want to know CPU usage is code segments, use *get_cpu_time*
> <https://www.ncl.ucar.edu/Document/Functions/Built-in/get_cpu_time.shtml>
>
> begTime = *get_cpu_time*()
>
> 68 wrf_prec = new((/n_years-1,n_hr_season/),float)
> 69 ; k is loop counter
> 70 k = 1
> 71 do i = 0, n_years - 2
> 72 temp =
> dimsizes(f_in[:]->PREC_ACC_NC(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2))
> 73 wrf_prec(i,0:temp-1) =
> f_in[:]->PREC_ACC_NC(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2)
> 74 delete(temp)
> 75 print(k)
> 76 k = k + 1
> 77 end do
>
> print("Loop Time: " + (*get_cpu_time*() - begTime+ " seconds")
> =====
>
> [2] The following requires NCL to invoke the underlying netCDF software.
>
> f_in*[:]*->PREC_ACC_NC(wrf_s_ind(i)*:*wrf_e_ind(i),nm1,nm2)
>
> For each 'i' [year] **ALL** files are scanned to extract in small temporal
> subset at one location [nm1,nm2]. There is no 'magic'. Under the hood, each
> invocation of the above requires the underlying software to index into the
> appropriate files. netCDF software is robust but may not be efficient for
> this repetitive type of application.
>
> [3] You are compounding the inefficiency by doing the above operation *twice!
> *The following uses NCL's reassignment syntax to allow for the
> possibility of different sized 'time' dimensions for each iteration.
>
> begTime = *get_cpu_time*()
> do i = 0, n_years - 2
> work* :=* f_in[:]->PREC_ACC_NC(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2) ; size 'ntim'
>
> temp := dimsizes(work) ; := likely not
> necessary here
> wrf_prec(i,0:temp-1) = work
> ;;print("k="+k)
> k = k + 1
> end do
> print("Loop Time: " + (*get_cpu_time*() - begTime+ " seconds")
>
> [4] Likely, even faster would be
>
> nm = getind_latlon2d(lat2d,lon2d,a,b)
> nm1 = nm(0,0)
> nm2 = nm(0,1)
> ...
> ; in memory
> PRECIP = f[:]->PREC_ACC_NC(:,nm1,nm2) ; ===> PRECIP
> *(time) a 1D array*
> * printVarSummary*(PRECIP)
>
> Then rewrite the code for a 1D array.
> k = 1
> do i = 0, n_years - 2
> temp = dimsizes( PRECIP)
> wrf_prec(i,0:temp-1) = PRECIP(wrf_s_ind(i):wrf_e_ind(i))
> ;;print("k="+k)
> k = k + 1
> end do
> =====
> However, you could do a 'trick' to still use 3D
>
> PRECIP = f[:]->PREC_ACC_NC(:,nm1:nm1,nm2:nm2) ; ===> PRECIP(
> *time,1,1) 3D array*
> *printVarSummary*(PRECIP)
>
> *...*
> nm1 = 0 ; reset
> nm2 = 0
> ...
> k = 1
> do i = 0, n_years - 2
> temp = dimsizes( PRECIP)
> wrf_prec(i,0:temp*(0)*-1) =
> PRECIP(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2)
> ;;print("k="+k)
> k = k + 1
> end do
>
> Good luck
>
>
> On Tue, Aug 6, 2019 at 10:12 PM Rick Brownrigg via ncl-talk <
> ncl-talk at ucar.edu> wrote:
>
>> OK, hopefully someone else can weigh in here -- I don't see anything
>> auspicious about lines 70-77, but I'm not expert with those data...
>>
>>
>> On Tue, Aug 6, 2019 at 8:50 PM Zach Rieck <zrr817 at gmail.com> wrote:
>>
>>> Thanks Rick!
>>>
>>> I think the thing that slows it down most is the do loop from 70-77 for
>>> some reason. Not sure if they arguments I'm using could be more specific to
>>> avoid pulling in so much data. Hard to tell, and I agree I wish that NCL
>>> was more efficient with that! Us programmers who are used to doing loops
>>> really need them.
>>>
>>> I got the divergence issue figure out, just had a -1 inside a
>>> parenthesis when it should have been outside.
>>>
>>> -Zach Rieck
>>> zrr817 at gmail.com
>>> (513)-502-5652
>>>
>>>
>>> On Tue, Aug 6, 2019 at 8:36 PM Rick Brownrigg <brownrig at ucar.edu> wrote:
>>>
>>>> Hi Zach,
>>>>
>>>> Just a comment and a conjecture: NCL's string processing is
>>>> notoriously inefficient when the number of strings involved gets large
>>>> (something I'd dearly loved to have fix, but...). With that in mind, I
>>>> wonder if the string processing around lines 22-41 might be the problem, in
>>>> which you convert the variable Time coming out of the WRF files to strings,
>>>> and then use the str_get_cols() function to pick out pieces of
>>>> year/month/day/etc.
>>>>
>>>> I don't really know much about it, but as far as I can tell, the Time
>>>> variable from a WRF file is character[*][*]? Converting sequences of
>>>> characters to integers does not seem to be directly supported, but you
>>>> might try something like:
>>>>
>>>> ncl 1> time = (/50C,48C,49C,57C, 48C, 56C, 48C, 54C/) ; funky notation
>>>> for character literals; this should be today's date: 20190806
>>>> ncl 2> print("year = " + tointeger(str_get_cols(time,0,3)))
>>>> ncl 3> print("mon = " + tointeger(str_get_cols(time,4,5)))
>>>> ncl 4> print("day = " + tointeger(str_get_cols(time,6,7)))
>>>>
>>>> This works, but I don't know if its an improvement; it issues warnings
>>>> to the effective of:
>>>>
>>>> warning:Argument 0 of the current function or procedure was coerced to
>>>> the appropriate type and thus will not change if the function or procedure
>>>> modifies its value
>>>>
>>>> I don't know if, under the hood, "coercion" is more efficient than
>>>> straight up conversion.
>>>>
>>>> I am not qualified to address the science question about the divergence
>>>> of the curves.
>>>>
>>>> Good luck!
>>>> Rick
>>>>
>>>>
>>>> On Tue, Aug 6, 2019 at 5:50 PM Zach Rieck via ncl-talk <
>>>> ncl-talk at ucar.edu> wrote:
>>>>
>>>>> To Whom it May Concern-
>>>>>
>>>>> I have been having an issue with a plot I have generated with this
>>>>> code file, and I need to be able to quickly troubleshoot the data it is
>>>>> pulling in. My code seems to be doing exactly what I want, but I'm curious
>>>>> if there's anyway to make it run faster/more efficiently? I know this may
>>>>> difficult since I'm having to skip over a bunch of time periods and
>>>>> organize a lot of data into different sized groups, so if the answer is I
>>>>> can't, that's fine.
>>>>>
>>>>> I've attached the figure in question as well just in case you see
>>>>> anything that might be causing the spread near the top of the figure. I've
>>>>> gone through my code very carefully, and I'm pretty sure it's an issue with
>>>>> the dataset, but if you notice anything else I can try fixing please let me
>>>>> know.
>>>>>
>>>>> Thanks in advance for the help!
>>>>>
>>>>> Respectfully,
>>>>>
>>>>> -Zach Rieck
>>>>> zrr817 at gmail.com
>>>>> (513)-502-5652
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190809/52b7f568/attachment.html>
More information about the ncl-talk
mailing list