[ncl-talk] optimization problem

Wed Jul 27 11:13:44 MDT 2016

If N total number of observations, then maybe

    pdf2 = *pdfxy*(x, y, 0, 0, False)
    n    = toint( N*pdf2 )   ; truncate

likely, n < N (but close)

Also:
http://www.ncl.ucar.edu/Document/Functions/Built-in/round.shtml

       n = round( N*pdf2, 3 )

You could do it 'directly'

       n = round( N**pdfxy*(x, y, 0, 0, False), 3 )

likely n.ne.N but it may be close enough.

Of course, under-the-hood, pdfxy is doing bin-counts.
However, prior to returning, the counts are divided by N'

where N' is the number of non-missing values.

D

On Tue, Jul 26, 2016 at 3:32 PM, Marston Johnston <shejo284 at gmail.com>
wrote:

> Hi Mary,
>
> Thanks for looking at this. Unfortunately the idea you put forth doesn't
> work.
> I played around with it and it was my thought to play around with the
> "where" statement, but the problem is that I need to find all the points in mhs5_2d
> = a single value in char AND all the points tb5m4_2d = to a single value
> in darr. The way you did it doesn't work for the intended purpose, which is
> to produce an output similar to PDFXY, but returning the counts (sum)
> instead of a procent or fraction.
> Originally, mhm(4,:,:) doesn't have the same dimensions as darr and char,
> but I can easily change this. However, this doesn't solve the problem.
> If we take PDFXY as a template, how do I get PDFXY to return the counts?
>
> /M
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Marston S. Johnston, PhD
> Department of Earth Sciences
> University of Gothenburg, Sweden
> Email: marston.johnston at gu.se
> Phone: +46-31-7864901
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Only the fruitful thing is true!
>
> On Tue, Jul 26, 2016 at 6:55 PM, Mary Haley <haley at ucar.edu> wrote:
>
>> Marston,
>>
>> In general, I recommend trying to use array arithmetic over looping, even
>> if it means using extra memory to create 2D versions of 1D arrays.
>>
>> If you have enough memory to create 2D arrays of your various 1D arrays,
>> then I recommend replacing all of the code after the last "copy_VarMeta"
>> call with the following (UNTESTED) code:
>>
>> I'm not exactly sure I have all the dimensions of your various arrays
>> correct, so this code may need some correction. For example, I don't know
>> what the dimensions of "mhs" are, so I'm not sure that I have that part
>> right.  If mhs is already dimensioned ndarr x nchar, then the mhsx_2d lines
>> should look simply like this:
>>
>>   mhs5_2d := round(mhs(4,:,:),0)
>>
>>
>> ; Create 2D versions of these arrays for faster (?) calculations later
>>
>> dims_2d = (/ndarr,nchar/)
>> mhs5_2d = conform_dims(dims_2d,round(ndtooned(mhs(4,:,:)),0),1)
>> mhs4_2d = conform_dims(dims_2d,round(ndtooned(mhs(3,:,:)),0),1)
>> mhs3_2d = conform_dims(dims_2d,round(ndtooned(mhs(2,:,:)),0),1)
>>
>> tb5m4_2d = conform_dims(dims_2d,round(ndtooned(tb5m4),0),0)
>> tb5m3_2d = conform_dims(dims_2d,round(ndtooned(tb5m3),0),0)
>> tb4m3_2d = conform_dims(dims_2d,round(ndtooned(tb4m3),0),0)
>>
>> darr_2d = conform_dims(dims_2d,darr,0)
>> char_2d = conform_dims(dims_2d,char,1)
>>
>> ; This creates a 2D logical array of Trues/Falses
>> mhs5_eq_char = mhs5_2d .eq. char_2d
>>
>> ; Create 2D arrays containing 1s and 0s
>> tmp5 = where(mhs5_eq_char .and. tb5m4_2d .eq. darr_2d, 1, 0)
>> tmp4 = where(mhs5_eq_char .and. tb5m3_2d .eq. darr_2d, 1, 0)
>> tmp3 = where(mhs5_eq_char .and. tb4m3_2d .eq. darr_2d, 1, 0)
>>
>> hist5m4 = hist5m4 + tmp5
>> hist5m3 = hist5m3 + tmp4
>> hist4m3 = hist4m3 + tmp3
>>
>> delete([/mhs5_eq_char,
>> mhs5_2d,mhs4_2d,mhs3_2d,darr_2d,char_2d,tmp5,tmp4,tmp3/])
>>
>> On Tue, Jul 26, 2016 at 6:05 AM, Marston Johnston <shejo284 at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm wondering if there is a way to speed up the do loops in the code
>>> below?
>>> I'm not sure it can be done, but I'm not an expert.
>>>
>>> ;***** 2D histogram ******
>>> ; x and y axis
>>> darr = ispan(-80,80,1)
>>> sarr = new(dimsizes(darr),string)
>>> sarr = " "
>>> char = ispan(150,300,1)
>>>
>>> ; histogram bins
>>> hist5m4 = new((/dimsizes(darr),dimsizes(char)/),float)
>>> hist5m4 at _FillValue = default_fillvalue("float")
>>> hist5m4!0 = "diff"
>>> hist5m4!1 = "Tb"
>>> hist5m4&diff = darr
>>> hist5m4&Tb = char
>>> ndarr = dimsizes(darr)
>>> nchar = dimsizes(char)
>>> hist5m4 = 0.0
>>> hist5m3 = hist5m4
>>> hist5m3 = 0.0
>>> hist4m3 = hist5m4
>>> hist4m3 = 0.0
>>>
>>> tb5m4 = mhs(4,:,:) - mhs(3,:,:)
>>> copy_VarMeta(mhs(3,:,:),tb5m4)
>>> tb5m3 = mhs(4,:,:) - mhs(2,:,:)
>>> copy_VarMeta(mhs(2,:,:),tb5m3)
>>> tb4m3 = mhs(3,:,:) - mhs(2,:,:)
>>> copy_VarMeta(mhs(2,:,:),tb5m3)
>>>
>>> mhs5 := round(ndtooned(mhs(4,:,:)),0)
>>> mhs4 := round(ndtooned(mhs(3,:,:)),0)
>>> mhs3 := round(ndtooned(mhs(2,:,:)),0)
>>>
>>> tb5m4 := round(ndtooned(tb5m4),0)
>>> tb5m3 := round(ndtooned(tb5m3),0)
>>> tb4m3 := round(ndtooned(tb4m3),0)
>>>
>>> ; print(darr(y)+" <= DARR:CH5AR => "+ch5ar(x))
>>> ; Fill the bins
>>> do x = 0, nchar - 1
>>>    do y = 0, ndarr - 1
>>>       tmp := ind(mhs5.eq.char(x) .and. tb5m4.eq.darr(y))
>>>       if(dimsizes(tmp).gt.1) then
>>>         hist5m4(y,x) = hist5m4(y,x) + (dimsizes(tmp) - 1)
>>>       end if
>>>       tmp := ind(mhs5.eq.char(x) .and. tb5m3.eq.darr(y))
>>>       if(dimsizes(tmp).gt.1) then
>>>         hist5m3(y,x) = hist5m3(y,x) + (dimsizes(tmp) - 1)
>>>       end if
>>>       tmp := ind(mhs4.eq.char(x) .and. tb4m3.eq.darr(y))
>>>       if(dimsizes(tmp).gt.1) then
>>>         hist4m3(y,x) = hist4m3(y,x) + (dimsizes(tmp) - 1)
>>>       end if
>>>   end do
>>> end do
>>>
>>> Appreciate your help with this,
>>> /M
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> Marston S. Johnston, PhD
>>> Department of Earth Sciences
>>> University of Gothenburg, Sweden
>>> Email: marston.johnston at gu.se
>>> Phone: +46-31-7864901
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> Only the fruitful thing is true!
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160727/387efd9d/attachment.html 