[ncl-talk] "graphic" array cannot store many figures?

[music piano]

Thanks, Dave. You are right.
 "graphic array" do consume a lot of memory. After following your
suggestion to reduce the size of the array, my code works!

On Fri, Nov 10, 2017 at 7:25 PM, Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> Graphic arrays in NCL consume large amounts of memory in hidden
> structures.  More memory is consumed with each plot command.  It is likely
> that your script is running out of memory or slowing way down because of
> lots of virtual memory swapping.
>
> I agree with most of Alan's suggestions.  However I think it is not
> necessary in this case to break up the processing and plotting.  The most
> important change would be to restructure your loops so that you are not
> creating such a large graphics array.  I have successfully made large PDF
> files with hundreds of panel plot frames, using a careful loop structure
> that only reads, processes, and plots one PDF frame at a time.
>
> Will a new structure like this work for you?  Notice that the smaller 2-D
> graphics array gets reused at every time step, which generally means that
> the hidden memory is reclaimed each time.  This assumes there are no
> calculations between time steps.  This might not be true.  However there
> are other solutions for that case, if needed.
>
> plot_1_2d=new((/num_var,nlevels/),"graphic")
>
> do it=0, ntimes-1
>
>      do var_id=num_var-1
>               do level=nlevels-1
>                        ........................... ; I skip my code here
>                        plot_1_2d(var_id,level) =
> contour_data_1_2d(it,var_id,level)
>               end do
>      end do
>
>      do var_id=num_var-1
>               do level=nlevels-1
>                       gsn_panel(wks,(/plot_1_2d(var_id,level)/),(/1,1/),resP)
>
>                       frame(wks)    ; ???
>               end do
>      end do
>
> end do
>
> --Dave
>
>
> On Fri, Nov 10, 2017 at 4:21 PM, music piano <musicpianoljy at gmail.com>
> wrote:
>
>> Hi,  everyone
>>
>> Sorry for bothering you. Thanks for Alan's detail suggestions. I like
>> your idea that process the data before plotting. I didn't show my original
>> code which is very lengthy, in which I actually process the data inside ncl
>> since it has convenient functions to deal with WRF data, such as
>> interpolating to pressure levels.
>>
>> The code I show here is just a structure to demonstrate the problem, in
>> which I made a typo error(should be level, not k). I actually use panel
>> plot to organize my figure, not just drawing one figure as I showed in this
>> email. Thus, drawing before the panel plot is not suitable. That is why I
>> need an "graphic array" to store all the figures.
>>
>> The code actually stops before the panel plot. So I think it is not the
>> problem of panel plot. I am still interested to know why the script stops
>> working when I increase the number of loops.
>>
>> Anybody has more suggestions?
>>
>> Best regards
>> Richard
>>
>>
>> On Fri, Nov 10, 2017 at 2:41 PM, Alan Brammer <abrammer at albany.edu>
>> wrote:
>>
>>> There might be some copy/paste errors in your email and I don't know
>>> about limits on graphic array sizes but a couple comments before someone
>>> from the NCL team can reply.
>>>
>>> Possibly most importantly, given that you're drawing individual plots,
>>> why not just plot in the first loop and drop the dependence on the array.
>>>
>>> Plots have a bunch of attributes connected to them I feel like using the
>>> (/ /) notation will likely disrupt how NCL handles graphics and tickmarks
>>> etc.  Although if this works on smaller loops maybe it doesn't matter.
>>>
>>> Why use gsn_panel() for a single plot? Why not just draw() and frame(),
>>> and why frame() after gsn_panel() instead of just letting gsn_panel() do
>>> it.
>>>
>>> Your 2nd nested loop uses level as the iterator but you use k as the
>>> index in the gsn_panel call.  (copy paste error or actual typo?)
>>>
>>> Finally, if you're processing 650GB of data, it's probably best to
>>> process the data and save the processed output to a netcdf. Then have a
>>> second script that plots the processed data. This way you can play with the
>>> plotting criteria without having to run through all of the data again.
>>>
>>> Or if you're just digging through raw data which based on the loops you
>>> may well be doing, panoply, ncview or similar would allow you to just
>>> visualise the data without having to process it at all. Are you sure you
>>> want 972 pages in a single pdf?
>>>
>>> Good luck,
>>>
>>> Alan
>>>
>>>
>>> On Fri, Nov 10, 2017 at 2:17 PM, music piano <musicpianoljy at gmail.com>
>>> wrote:
>>>
>>>> Hi, everyone
>>>>
>>>> I produced many figures and passed to panel plot. I define a "graphic"
>>>> array to store my figures.
>>>>
>>>> The following script is the essential part to produce the figure. There
>>>> are many files, the code read one file at a time. It loops over many times,
>>>> many variables and many vertical levels.
>>>>
>>>> After all the figures were stored in plot_1_3d array. the panel plot
>>>> plot them. *This code works fine* when plot_1_3d is a smaller array.
>>>> For instance, ntimes=4, num_var=2, nlevels=18.
>>>>
>>>>  However, when I loop over many files, such as ntimes=27, the program
>>>> stops without any error message when it loops to it=12 or some other
>>>> number. Each of the file the program read is 25GB, but I delete old
>>>> variables in every time loop. The only array exists over the whole period
>>>> is the "graphic" array. I am not sure it is the memory issue. I run this
>>>> code on a supercomputer node, which has  32GB memory. The output pdf file
>>>> for the figure is 43MB if ntimes=4, nlevels=18, num_var=2.
>>>>
>>>> I will be very appreciated if anybody can help me.
>>>>
>>>> My scripts:
>>>> plot_1_3d=new((/ntimes,num_var,nlevels/),"graphic")
>>>>
>>>> do it=0, ntimes-1
>>>>     do var_id=num_var-1
>>>>               do level=nlevels-1
>>>>                          ........................... ; I skip my code
>>>> here
>>>>                         plot_1_3d(it,var_id,level) =
>>>> contour_data_1_3d(var_id,level)
>>>>                end do
>>>>      end do
>>>> end do
>>>>
>>>> do it=0, ntimes-1
>>>>     do var_id=num_var-1
>>>>               do level=nlevels-1
>>>>                       gsn_panel(wks,(/plot_1_3d(it,var_id,k)/),(/1,1/),resP
>>>>     )
>>>>                        frame(wks)
>>>>       end do
>>>>      end do
>>>> end do
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171111/7ccde953/attachment.html>