[ncl-talk] "graphic" array cannot store many figures?

[Dave Allured - NOAA Affiliate]

Graphic arrays in NCL consume large amounts of memory in hidden
structures.  More memory is consumed with each plot command.  It is likely
that your script is running out of memory or slowing way down because of
lots of virtual memory swapping.

I agree with most of Alan's suggestions.  However I think it is not
necessary in this case to break up the processing and plotting.  The most
important change would be to restructure your loops so that you are not
creating such a large graphics array.  I have successfully made large PDF
files with hundreds of panel plot frames, using a careful loop structure
that only reads, processes, and plots one PDF frame at a time.

Will a new structure like this work for you?  Notice that the smaller 2-D
graphics array gets reused at every time step, which generally means that
the hidden memory is reclaimed each time.  This assumes there are no
calculations between time steps.  This might not be true.  However there
are other solutions for that case, if needed.

plot_1_2d=new((/num_var,nlevels/),"graphic")

do it=0, ntimes-1

     do var_id=num_var-1
              do level=nlevels-1
                       ........................... ; I skip my code here
                       plot_1_2d(var_id,level) =
contour_data_1_2d(it,var_id,level)
              end do
     end do

     do var_id=num_var-1
              do level=nlevels-1
                      gsn_panel(wks,(/plot_1_2d(var_id,level)/),(/1,1/),resP)

                      frame(wks)    ; ???
              end do
     end do

end do

--Dave


On Fri, Nov 10, 2017 at 4:21 PM, music piano <musicpianoljy at gmail.com>
wrote:

> Hi,  everyone
>
> Sorry for bothering you. Thanks for Alan's detail suggestions. I like your
> idea that process the data before plotting. I didn't show my original code
> which is very lengthy, in which I actually process the data inside ncl
> since it has convenient functions to deal with WRF data, such as
> interpolating to pressure levels.
>
> The code I show here is just a structure to demonstrate the problem, in
> which I made a typo error(should be level, not k). I actually use panel
> plot to organize my figure, not just drawing one figure as I showed in this
> email. Thus, drawing before the panel plot is not suitable. That is why I
> need an "graphic array" to store all the figures.
>
> The code actually stops before the panel plot. So I think it is not the
> problem of panel plot. I am still interested to know why the script stops
> working when I increase the number of loops.
>
> Anybody has more suggestions?
>
> Best regards
> Richard
>
>
> On Fri, Nov 10, 2017 at 2:41 PM, Alan Brammer <abrammer at albany.edu> wrote:
>
>> There might be some copy/paste errors in your email and I don't know
>> about limits on graphic array sizes but a couple comments before someone
>> from the NCL team can reply.
>>
>> Possibly most importantly, given that you're drawing individual plots,
>> why not just plot in the first loop and drop the dependence on the array.
>>
>> Plots have a bunch of attributes connected to them I feel like using the
>> (/ /) notation will likely disrupt how NCL handles graphics and tickmarks
>> etc.  Although if this works on smaller loops maybe it doesn't matter.
>>
>> Why use gsn_panel() for a single plot? Why not just draw() and frame(),
>> and why frame() after gsn_panel() instead of just letting gsn_panel() do
>> it.
>>
>> Your 2nd nested loop uses level as the iterator but you use k as the
>> index in the gsn_panel call.  (copy paste error or actual typo?)
>>
>> Finally, if you're processing 650GB of data, it's probably best to
>> process the data and save the processed output to a netcdf. Then have a
>> second script that plots the processed data. This way you can play with the
>> plotting criteria without having to run through all of the data again.
>>
>> Or if you're just digging through raw data which based on the loops you
>> may well be doing, panoply, ncview or similar would allow you to just
>> visualise the data without having to process it at all. Are you sure you
>> want 972 pages in a single pdf?
>>
>> Good luck,
>>
>> Alan
>>
>>
>> On Fri, Nov 10, 2017 at 2:17 PM, music piano <musicpianoljy at gmail.com>
>> wrote:
>>
>>> Hi, everyone
>>>
>>> I produced many figures and passed to panel plot. I define a "graphic"
>>> array to store my figures.
>>>
>>> The following script is the essential part to produce the figure. There
>>> are many files, the code read one file at a time. It loops over many times,
>>> many variables and many vertical levels.
>>>
>>> After all the figures were stored in plot_1_3d array. the panel plot
>>> plot them. *This code works fine* when plot_1_3d is a smaller array.
>>> For instance, ntimes=4, num_var=2, nlevels=18.
>>>
>>>  However, when I loop over many files, such as ntimes=27, the program
>>> stops without any error message when it loops to it=12 or some other
>>> number. Each of the file the program read is 25GB, but I delete old
>>> variables in every time loop. The only array exists over the whole period
>>> is the "graphic" array. I am not sure it is the memory issue. I run this
>>> code on a supercomputer node, which has  32GB memory. The output pdf file
>>> for the figure is 43MB if ntimes=4, nlevels=18, num_var=2.
>>>
>>> I will be very appreciated if anybody can help me.
>>>
>>> My scripts:
>>> plot_1_3d=new((/ntimes,num_var,nlevels/),"graphic")
>>>
>>> do it=0, ntimes-1
>>>     do var_id=num_var-1
>>>               do level=nlevels-1
>>>                          ........................... ; I skip my code
>>> here
>>>                         plot_1_3d(it,var_id,level) =
>>> contour_data_1_3d(var_id,level)
>>>                end do
>>>      end do
>>> end do
>>>
>>> do it=0, ntimes-1
>>>     do var_id=num_var-1
>>>               do level=nlevels-1
>>>                       gsn_panel(wks,(/plot_1_3d(it,var_id,k)/),(/1,1/),resP
>>>     )
>>>                        frame(wks)
>>>       end do
>>>      end do
>>> end do
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171110/111704eb/attachment.html>