[ncl-talk] NCL dynamic memory allocation error
Dennis Shea
shea at ucar.edu
Sun Aug 11 07:01:33 MDT 2019
A quick look at:
/glade/work/burakows/plot/NCL/MODIS_VernalWindow/MOD10A1
ncl 0> fils = systemfunc("ls MOD10A1*hdf")
ncl 1> nfils = dimsizes(fils)
ncl 2> print(nfils)
There are *17097* hdf files.
However, you want: 'yr=2012' .... I suggest that you import data from
that year only.
ncl 0> yr = 2012
ncl 1> fils = systemfunc("ls MOD10A1.A"*+**yr**+*"*.hdf") ;
all files [v04 and v05]
ncl 2> nfils = dimsizes(fils)
ncl 3> print(nfils)
There are *1265* hdf files.
---
Your 'snowLast.ncl' function is read protected so I can not test the script.
---
;;;;yrlist = ispan(2003,2016,1)
yrlist = ispan(fyr(0),fyr(nflist-1),1)
Replace
temp_flist = ind(fyr.eq.yr .and. hlist.eq.11 .and. vlist.eq.4)
With
temp_flist = ind(hlist.eq.11 .and. vlist.eq.4)
The above will save lots of memory because the original variable is
*[2400,2400].
*Specifically:
* YDim_MOD_Grid_Snow_500m = 2400 XDim_MOD_Grid_Snow_500m
= 2400*
f =
addfiles(dir+flist(temp_flist)+".he2","r")
ListSetType(f,"join")
SnowCover_byte = f[:]->$var$ ; *original data* from
*17097* files
printVarSummary(SnowCover_byte) ; (17097,2400,2400)
After inputting data from yr=2012 only
SnowCover_byte = f[:]->$var$ ; *2012 data* only from *1265
*files
printVarSummary(SnowCover_byte) ; (1265,2400,2400)
Further, *you know the desired regional indices. *
These should be used to further reduce the sizes by* importing only the
desired regional subset*
jStrt = ij(0,0) ; clarity
jLast = ij(0,1)
iStrt = ij(0,1)
iLast = ij(1,1)
SnowCover_byte = f[:]->$var$(jStrt:jLast,iStrt:iLast)
; *2012
data* only
printVarSummary(SnowCover_byte) ; look at this
*You can keep the data in 'byte' form.*
NCL's averaging function(s) will compute the averages correctly.
More memory savings because the following are *NOT *needed.
SnowCover_byt :=
SnowCover_byte(:,ij(0,0):ij(0,1),ij(0,1):ij(1,1))
SnowCover = byte2flt(SnowCover_byte) ; BIG memory
expansion [4x]
SnowCover = where(SnowCover.lt.0, SnowCover@
_FillValue,SnowCover)
In fact, just use
SnowCover = f[:]->$var$(jStrt:jLast,iStrt:iLast) ; *2012
subset data* only
The above methodology can be applied to the other subsetting sections.
On Sat, Aug 10, 2019 at 5:08 PM Burakowski, Elizabeth via ncl-talk <
ncl-talk at ucar.edu> wrote:
> Hello Again,
>
> I updated the script and it is running much faster, but I'm still getting
> the same errors as before:
>
> fatal:MapV41DHInitialize: dynamic memory allocation error:[errno=12]
> fatal:MapV41DHInitialize: error initializing map outline records
> ERROR 2 IN MPLNRI/MDLNRI - Can't open the ".names" file
> fatal:MapSetTrans: error initializing map: MAPINT - UNCLEARED PRIOR ERROR
> warning:["SetValues.c":474]:SetValuesChild never occurred on
> map.PlotManager: Error in mapPlotClass
> Segmentation fault (core dumped)
>
> The variable I'm plotting is only 400 KB and I've deleted every other
> unnecessary variable in the script. I'm beginning to think this is *not*
> a memory allocation error, but I still don't know what it is.
>
> For what it's worth, I have a nearly identical script that plots MODIS
> vegetation green-up instead of snow. The MODIS green-up script produces
> plots just fine, there are no errors. I cannot find any differences in the
> two scripts that would explain this.
>
> Green-Up script on Cheyenne:
> /glade/work/burakows/plot/NCL/MODIS_VernalWindow/plotMCD12Q2_Year.ncl
>
> Snow script on Cheyenne:
> /glade/work/burakows/plot/NCL/MODIS_VernalWindow/plotMOD10A1_Year.ncl
>
> I'm passing this error on to CISL to see if they can figure out what is
> going on.
>
> Thanks,
>
> Liz.
>
>
> Dr. Elizabeth A Burakowski
> She/Her/Hers
> Research Assistant Professor
> Earth Systems Research Center &
> Institute for the Study of Earth, Oceans, and Space
> University of New Hampshire
> 8 College Road Durham, NH 03824
> 603-862-1796
> elizabeth.burakowski at unh.edu
> ------------------------------
> *From:* Burakowski, Elizabeth <Elizabeth.Burakowski at unh.edu>
> *Sent:* Saturday, August 10, 2019 10:35 AM
> *To:* Rick Brownrigg <brownrig at ucar.edu>
> *Cc:* ncl-talk at ucar.edu <ncl-talk at ucar.edu>
> *Subject:* Re: [ncl-talk] NCL dynamic memory allocation error
>
> Hi Rick,
>
> I'm running the script on Casper, not my personal computer. I've been
> using the delete function to delete every variable not need for plotting
> along the way. I'll try requesting more memory on Casper. Looks like the
> default is 1.87 GB per core.
>
> Thanks,
>
> Liz
>
>
> Dr. Elizabeth A Burakowski
> She/Her/Hers
> Research Assistant Professor
> Earth Systems Research Center &
> Institute for the Study of Earth, Oceans, and Space
> University of New Hampshire
> 8 College Road Durham, NH 03824
> 603-862-1796
> elizabeth.burakowski at unh.edu
> ------------------------------
> *From:* Rick Brownrigg <brownrig at ucar.edu>
> *Sent:* Saturday, August 10, 2019 10:29 AM
> *To:* Burakowski, Elizabeth <Elizabeth.Burakowski at unh.edu>
> *Cc:* ncl-talk at ucar.edu <ncl-talk at ucar.edu>
> *Subject:* Re: [ncl-talk] NCL dynamic memory allocation error
>
> *Caution - External Email*
> ------------------------------
> Hi Elizabeth,
>
> errno=12 is indeed an "unable to allocate memory" error on linux/MacOS
> systems. Why it manifests in the mapping codes is unclear, but that does
> sort of suggests that the memory depletion is happening late in the game;
> i.e., it sounds like you've successfully read and processed the data and
> are at the final plotting stage. You might try using the NCL delete()
> function to free up memory used by variables that are no longer needed for
> plotting. Another suggestion would be to shutdown any large
> memory-footprint applications that may be running on your system at the
> time (e.g., browsers, wordprocessors, etc.)
>
> Hope that helps...
> Rick
>
> On Sat, Aug 10, 2019 at 7:49 AM Burakowski, Elizabeth via ncl-talk <
> ncl-talk at ucar.edu> wrote:
>
> Hello,
>
> I keep getting an error while trying to run an NCL script on casper
> (execdav):
>
> fatal:MapV41DHInitialize: dynamic memory allocation error:[errno=12]
> fatal:MapV41DHInitialize: error initializing map outline records
> ERROR 2 IN MPLNRI/MDLNRI - Can't open the ".names" file
> fatal:MapSetTrans: error initializing map: MAPINT - UNCLEARED PRIOR ERROR
> warning:["SetValues.c":474]:SetValuesChild never occurred on
> map.PlotManager: Error in mapPlotClass
> Segmentation fault (core dumped)
>
> It is a rather memory intensive script (500 m MODIS data), but the
> ".names" file error 2 is throwing me off. I've successfully plotted other
> MODIS data using a similar script, so I'm not sure whats different about
> this one.
>
> Script is located here:
>
> /glade/u/home/burakows/work/plot/NCL/MODIS_VernalWindow/plotMOD10A1_Year.ncl
>
> Thank you,
>
> liz.
>
>
>
> Dr. Elizabeth A Burakowski
> She/Her/Hers
> Research Assistant Professor
> Earth Systems Research Center &
> Institute for the Study of Earth, Oceans, and Space
> University of New Hampshire
> 8 College Road Durham, NH 03824
> 603-862-1796
> elizabeth.burakowski at unh.edu
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ucar.edu_mailman_listinfo_ncl-2Dtalk&d=DwMFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=JW2rHHzYnVYi6ZDBI-2JFMNSreDPLZO-wwOrczEnOlg&m=dFK61NjWI9bYpf7Dj8JN1OK938G-xKdMzPnKw3vGa_g&s=b0aCHogAHmXT0ALNB7zGXNGY-s_TPZ4Xxxnf9jJN0Mw&e=>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190811/ae02d0ea/attachment.html>
More information about the ncl-talk
mailing list