[ncl-talk] Code Compiling Extremely Slowly
Dennis Shea
shea at ucar.edu
Wed Aug 7 10:05:28 MDT 2019
[1] If you want to know CPU usage is code segments, use *get_cpu_time*
<https://www.ncl.ucar.edu/Document/Functions/Built-in/get_cpu_time.shtml>
begTime = *get_cpu_time*()
68 wrf_prec = new((/n_years-1,n_hr_season/),float)
69 ; k is loop counter
70 k = 1
71 do i = 0, n_years - 2
72 temp =
dimsizes(f_in[:]->PREC_ACC_NC(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2))
73 wrf_prec(i,0:temp-1) =
f_in[:]->PREC_ACC_NC(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2)
74 delete(temp)
75 print(k)
76 k = k + 1
77 end do
print("Loop Time: " + (*get_cpu_time*() - begTime+ " seconds")
=====
[2] The following requires NCL to invoke the underlying netCDF software.
f_in*[:]*->PREC_ACC_NC(wrf_s_ind(i)*:*wrf_e_ind(i),nm1,nm2)
For each 'i' [year] **ALL** files are scanned to extract in small temporal
subset at one location [nm1,nm2]. There is no 'magic'. Under the hood, each
invocation of the above requires the underlying software to index into the
appropriate files. netCDF software is robust but may not be efficient for
this repetitive type of application.
[3] You are compounding the inefficiency by doing the above operation *twice!
*The following uses NCL's reassignment syntax to allow for the possibility
of different sized 'time' dimensions for each iteration.
begTime = *get_cpu_time*()
do i = 0, n_years - 2
work* :=*
f_in[:]->PREC_ACC_NC(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2) ; size
'ntim'
temp := dimsizes(work) ; := likely not
necessary here
wrf_prec(i,0:temp-1) = work
;;print("k="+k)
k = k + 1
end do
print("Loop Time: " + (*get_cpu_time*() - begTime+ " seconds")
[4] Likely, even faster would be
nm = getind_latlon2d(lat2d,lon2d,a,b)
nm1 = nm(0,0)
nm2 = nm(0,1)
...
; in memory
PRECIP = f[:]->PREC_ACC_NC(:,nm1,nm2) ; ===> PRECIP
*(time) a 1D array*
* printVarSummary*(PRECIP)
Then rewrite the code for a 1D array.
k = 1
do i = 0, n_years - 2
temp = dimsizes( PRECIP)
wrf_prec(i,0:temp-1) = PRECIP(wrf_s_ind(i):wrf_e_ind(i))
;;print("k="+k)
k = k + 1
end do
=====
However, you could do a 'trick' to still use 3D
PRECIP = f[:]->PREC_ACC_NC(:,nm1:nm1,nm2:nm2) ; ===> PRECIP(
*time,1,1) 3D array*
*printVarSummary*(PRECIP)
*...*
nm1 = 0 ; reset
nm2 = 0
...
k = 1
do i = 0, n_years - 2
temp = dimsizes( PRECIP)
wrf_prec(i,0:temp*(0)*-1) =
PRECIP(wrf_s_ind(i):wrf_e_ind(i),nm1,nm2)
;;print("k="+k)
k = k + 1
end do
Good luck
On Tue, Aug 6, 2019 at 10:12 PM Rick Brownrigg via ncl-talk <
ncl-talk at ucar.edu> wrote:
> OK, hopefully someone else can weigh in here -- I don't see anything
> auspicious about lines 70-77, but I'm not expert with those data...
>
>
> On Tue, Aug 6, 2019 at 8:50 PM Zach Rieck <zrr817 at gmail.com> wrote:
>
>> Thanks Rick!
>>
>> I think the thing that slows it down most is the do loop from 70-77 for
>> some reason. Not sure if they arguments I'm using could be more specific to
>> avoid pulling in so much data. Hard to tell, and I agree I wish that NCL
>> was more efficient with that! Us programmers who are used to doing loops
>> really need them.
>>
>> I got the divergence issue figure out, just had a -1 inside a parenthesis
>> when it should have been outside.
>>
>> -Zach Rieck
>> zrr817 at gmail.com
>> (513)-502-5652
>>
>>
>> On Tue, Aug 6, 2019 at 8:36 PM Rick Brownrigg <brownrig at ucar.edu> wrote:
>>
>>> Hi Zach,
>>>
>>> Just a comment and a conjecture: NCL's string processing is notoriously
>>> inefficient when the number of strings involved gets large (something I'd
>>> dearly loved to have fix, but...). With that in mind, I wonder if the
>>> string processing around lines 22-41 might be the problem, in which you
>>> convert the variable Time coming out of the WRF files to strings, and then
>>> use the str_get_cols() function to pick out pieces of year/month/day/etc.
>>>
>>> I don't really know much about it, but as far as I can tell, the Time
>>> variable from a WRF file is character[*][*]? Converting sequences of
>>> characters to integers does not seem to be directly supported, but you
>>> might try something like:
>>>
>>> ncl 1> time = (/50C,48C,49C,57C, 48C, 56C, 48C, 54C/) ; funky notation
>>> for character literals; this should be today's date: 20190806
>>> ncl 2> print("year = " + tointeger(str_get_cols(time,0,3)))
>>> ncl 3> print("mon = " + tointeger(str_get_cols(time,4,5)))
>>> ncl 4> print("day = " + tointeger(str_get_cols(time,6,7)))
>>>
>>> This works, but I don't know if its an improvement; it issues warnings
>>> to the effective of:
>>>
>>> warning:Argument 0 of the current function or procedure was coerced to
>>> the appropriate type and thus will not change if the function or procedure
>>> modifies its value
>>>
>>> I don't know if, under the hood, "coercion" is more efficient than
>>> straight up conversion.
>>>
>>> I am not qualified to address the science question about the divergence
>>> of the curves.
>>>
>>> Good luck!
>>> Rick
>>>
>>>
>>> On Tue, Aug 6, 2019 at 5:50 PM Zach Rieck via ncl-talk <
>>> ncl-talk at ucar.edu> wrote:
>>>
>>>> To Whom it May Concern-
>>>>
>>>> I have been having an issue with a plot I have generated with this code
>>>> file, and I need to be able to quickly troubleshoot the data it is pulling
>>>> in. My code seems to be doing exactly what I want, but I'm curious if
>>>> there's anyway to make it run faster/more efficiently? I know this may
>>>> difficult since I'm having to skip over a bunch of time periods and
>>>> organize a lot of data into different sized groups, so if the answer is I
>>>> can't, that's fine.
>>>>
>>>> I've attached the figure in question as well just in case you see
>>>> anything that might be causing the spread near the top of the figure. I've
>>>> gone through my code very carefully, and I'm pretty sure it's an issue with
>>>> the dataset, but if you notice anything else I can try fixing please let me
>>>> know.
>>>>
>>>> Thanks in advance for the help!
>>>>
>>>> Respectfully,
>>>>
>>>> -Zach Rieck
>>>> zrr817 at gmail.com
>>>> (513)-502-5652
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk at ucar.edu
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190807/3387910f/attachment.html>
More information about the ncl-talk
mailing list