<div dir="ltr"><div>A quick look at:</div><div><br></div><div>/glade/work/burakows/plot/NCL/MODIS_VernalWindow/MOD10A1</div><div><br></div><div>ncl 0> fils = systemfunc("ls MOD10A1*hdf")<br>ncl 1> nfils = dimsizes(fils)<br>ncl 2> print(nfils)</div><div><br></div><div>There are <b>17097</b> hdf files.</div><div><br></div><div>However, you want: 'yr=2012' .... I suggest that you import data from that year only.</div><div><br></div><div>ncl 0> yr = 2012<br>ncl 1> fils = systemfunc("ls MOD10A1.A"<b>+</b><span style="color:rgb(0,0,255)"><b>yr</b></span><b>+</b>"*.hdf") ; all files [v04 and v05]<br>ncl 2> nfils = dimsizes(fils)<br>ncl 3> print(nfils)</div><div><br></div><div><div>There are <b>1265</b> hdf files.</div><div><br></div></div><div>---</div><div>Your 'snowLast.ncl' function is read protected so I can not test the script.</div><div>---</div><div><br></div><div>;;;;yrlist = ispan(2003,2016,1)<br> yrlist = ispan(fyr(0),fyr(nflist-1),1) <br></div><div><br></div><div>Replace</div><div> temp_flist = ind(fyr.eq.yr .and. hlist.eq.11 .and. vlist.eq.4)</div><div>With</div><div> temp_flist = ind(hlist.eq.11 .and. vlist.eq.4)</div><div><br></div><div>The above will save lots of memory because the original variable is <b>[2400,2400]. </b>Specifically:</div><div><b><br></b></div><div><b> YDim_MOD_Grid_Snow_500m = 2400<br> XDim_MOD_Grid_Snow_500m = 2400</b></div><div><br></div><div> f = addfiles(dir+flist(temp_flist)+".he2","r") <br> ListSetType(f,"join")</div><div><br></div><div> SnowCover_byte = f[:]->$var$ ; <b>original data</b> from <b>17097</b> files</div><div> printVarSummary(SnowCover_byte) ; (17097,2400,2400) <br></div><div><br></div><div>After inputting data from yr=2012 only</div><div><br></div><div><div> SnowCover_byte = f[:]->$var$ ; <b>2012 data</b> only from <b>1265 </b>files</div><div> printVarSummary(SnowCover_byte) ; (1265,2400,2400)<br></div><div><br></div></div><div>Further, <b>you know the desired regional indices. </b><br></div><div>These should be used to further reduce the sizes by<b> importing only the desired regional subset</b><br></div><div><br></div><div> jStrt = ij(0,0) ; clarity<br></div><div> jLast = ij(0,1)</div><div> iStrt = ij(0,1)</div><div> iLast = ij(1,1)<br></div><div><br></div><div><div><div> SnowCover_byte = f[:]->$var$(jStrt:jLast,iStrt:iLast) ; <b>2012 data</b> only <br></div><div> printVarSummary(SnowCover_byte) ; look at this<br></div><div><br></div><div><b>You can keep the data in 'byte' form.</b> <br></div><div>NCL's averaging function(s) will compute the averages correctly. <br></div><div>More memory savings because the following are <b>NOT </b>needed.<br></div><div><br></div></div></div> SnowCover_byt := SnowCover_byte(:,ij(0,0):ij(0,1),ij(0,1):ij(1,1))<br> SnowCover = byte2flt(SnowCover_byte) ; BIG memory expansion [4x]<br><div> SnowCover = where(SnowCover.lt.0, SnowCover@_FillValue,SnowCover)</div><div><br></div><div>In fact, just use</div><div> <br><div> SnowCover = f[:]->$var$(jStrt:jLast,iStrt:iLast) ; <b>2012 subset data</b> only <br></div></div><div><br></div><div>The above methodology can be applied to the other subsetting sections.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 10, 2019 at 5:08 PM Burakowski, Elizabeth via ncl-talk <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hello Again, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I updated the script and it is running much faster, but I'm still getting the same errors as before:
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="color:black;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif">
<span>fatal:MapV41DHInitialize: dynamic memory allocation error:[errno=12]<br>
</span>
<div>fatal:MapV41DHInitialize: error initializing map outline records<br>
</div>
<span> ERROR 2 IN MPLNRI/MDLNRI - Can't open the ".names" file</span><br>
</div>
<div style="color:black;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif">
<span>fatal:MapSetTrans: error initializing map: MAPINT - UNCLEARED PRIOR ERROR<br>
</span>
<div>warning:["SetValues.c":474]:SetValuesChild never occurred on map.PlotManager: Error in mapPlotClass<br>
</div>
<span>Segmentation fault (core dumped)</span></div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The variable I'm plotting is only 400 KB and I've deleted every other unnecessary variable in the script. I'm beginning to think this is
<i>not</i> a memory allocation error, but I still don't know what it is. <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
For what it's worth, I have a nearly identical script that plots MODIS vegetation green-up instead of snow. The MODIS green-up script produces plots just fine, there are no errors. I cannot find any differences in the two scripts that would explain this.
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Green-Up script on Cheyenne:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
/glade/work/burakows/plot/NCL/MODIS_VernalWindow/plotMCD12Q2_Year.ncl<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Snow script on Cheyenne:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
/glade/work/burakows/plot/NCL/MODIS_VernalWindow/plotMOD10A1_Year.ncl</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I'm passing this error on to CISL to see if they can figure out what is going on.
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Liz.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784Signature">
<div></div>
<div name="divtagdefaultwrapper">
<font face="Corbel, sans-serif">Dr. Elizabeth A Burakowski</font></div>
<div name="divtagdefaultwrapper">
<font face="Corbel, sans-serif">She/Her/Hers<br>
</font>
<div><font face="Corbel, sans-serif">Research Assistant Professor</font></div>
<div><font face="Corbel, sans-serif">Earth Systems Research Center & </font></div>
<div><font face="Corbel, sans-serif">Institute for the Study of Earth, Oceans, and Space</font></div>
<div><font face="Corbel, sans-serif">University of New Hampshire</font></div>
<div><font face="Corbel, sans-serif">8 College Road Durham, NH 03824</font></div>
<div><font face="Corbel, sans-serif">603-862-1796</font></div>
<div><font face="Corbel, sans-serif"><a href="mailto:elizabeth.burakowski@unh.edu" target="_blank">elizabeth.burakowski@unh.edu</a></font></div>
</div>
</div>
<div id="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Burakowski, Elizabeth <<a href="mailto:Elizabeth.Burakowski@unh.edu" target="_blank">Elizabeth.Burakowski@unh.edu</a>><br>
<b>Sent:</b> Saturday, August 10, 2019 10:35 AM<br>
<b>To:</b> Rick Brownrigg <<a href="mailto:brownrig@ucar.edu" target="_blank">brownrig@ucar.edu</a>><br>
<b>Cc:</b> <a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a> <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>><br>
<b>Subject:</b> Re: [ncl-talk] NCL dynamic memory allocation error</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Rick, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I'm running the script on Casper, not my personal computer. I've been using the delete function to delete every variable not need for plotting along the way. I'll try requesting more memory on Casper. Looks like the default is 1.87 GB per core.
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Liz</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784x_Signature">
<div></div>
<div name="x_divtagdefaultwrapper">
<font face="Corbel, sans-serif">Dr. Elizabeth A Burakowski</font></div>
<div name="x_divtagdefaultwrapper">
<font face="Corbel, sans-serif">She/Her/Hers<br>
</font>
<div><font face="Corbel, sans-serif">Research Assistant Professor</font></div>
<div><font face="Corbel, sans-serif">Earth Systems Research Center & </font></div>
<div><font face="Corbel, sans-serif">Institute for the Study of Earth, Oceans, and Space</font></div>
<div><font face="Corbel, sans-serif">University of New Hampshire</font></div>
<div><font face="Corbel, sans-serif">8 College Road Durham, NH 03824</font></div>
<div><font face="Corbel, sans-serif">603-862-1796</font></div>
<div><font face="Corbel, sans-serif"><a href="mailto:elizabeth.burakowski@unh.edu" target="_blank">elizabeth.burakowski@unh.edu</a></font></div>
</div>
</div>
<div id="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Rick Brownrigg <<a href="mailto:brownrig@ucar.edu" target="_blank">brownrig@ucar.edu</a>><br>
<b>Sent:</b> Saturday, August 10, 2019 10:29 AM<br>
<b>To:</b> Burakowski, Elizabeth <<a href="mailto:Elizabeth.Burakowski@unh.edu" target="_blank">Elizabeth.Burakowski@unh.edu</a>><br>
<b>Cc:</b> <a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a> <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>><br>
<b>Subject:</b> Re: [ncl-talk] NCL dynamic memory allocation error</font>
<div> </div>
</div>
<div><strong><font color="red">Caution - External Email</font></strong>
<hr>
<div dir="ltr">
<div>Hi Elizabeth,</div>
<div><br>
</div>
<div>errno=12 is indeed an "unable to allocate memory" error on linux/MacOS systems. Why it manifests in the mapping codes is unclear, but that does sort of suggests that the memory depletion is happening late in the game; i.e., it sounds like you've successfully
read and processed the data and are at the final plotting stage. You might try using the NCL delete() function to free up memory used by variables that are no longer needed for plotting. Another suggestion would be to shutdown any large memory-footprint applications
that may be running on your system at the time (e.g., browsers, wordprocessors, etc.)</div>
<div><br>
</div>
<div>Hope that helps...</div>
<div>Rick<br>
</div>
</div>
<br>
<div class="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784x_x_gmail_quote">
<div dir="ltr" class="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784x_x_gmail_attr">On Sat, Aug 10, 2019 at 7:49 AM Burakowski, Elizabeth via ncl-talk <<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>> wrote:<br>
</div>
<blockquote class="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hello, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I keep getting an error while trying to run an NCL script on casper (execdav):</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>fatal:MapV41DHInitialize: dynamic memory allocation error:[errno=12]<br>
</span>
<div>fatal:MapV41DHInitialize: error initializing map outline records<br>
</div>
<span> ERROR 2 IN MPLNRI/MDLNRI - Can't open the ".names" file</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>fatal:MapSetTrans: error initializing map: MAPINT - UNCLEARED PRIOR ERROR<br>
</span>
<div>warning:["SetValues.c":474]:SetValuesChild never occurred on map.PlotManager: Error in mapPlotClass<br>
</div>
<span>Segmentation fault (core dumped)</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
It is a rather memory intensive script (500 m MODIS data), but the ".names" file error 2 is throwing me off. I've successfully plotted other MODIS data using a similar script, so I'm not sure whats different about this one.
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Script is located here: <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
/glade/u/home/burakows/work/plot/NCL/MODIS_VernalWindow/plotMOD10A1_Year.ncl<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thank you, <br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
liz.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-1769667609169115574gmail-m_4027719881712832557gmail-m_7522512266280788802gmail-m_-4078435723532774784x_x_gmail-m_2598301679700832486Signature">
<div></div>
<div name="x_x_divtagdefaultwrapper"><font face="Corbel, sans-serif">Dr. Elizabeth A Burakowski</font></div>
<div name="x_x_divtagdefaultwrapper"><font face="Corbel, sans-serif">She/Her/Hers<br>
</font>
<div><font face="Corbel, sans-serif">Research Assistant Professor</font></div>
<div><font face="Corbel, sans-serif">Earth Systems Research Center & </font></div>
<div><font face="Corbel, sans-serif">Institute for the Study of Earth, Oceans, and Space</font></div>
<div><font face="Corbel, sans-serif">University of New Hampshire</font></div>
<div><font face="Corbel, sans-serif">8 College Road Durham, NH 03824</font></div>
<div><font face="Corbel, sans-serif">603-862-1796</font></div>
<div><font face="Corbel, sans-serif"><a href="mailto:elizabeth.burakowski@unh.edu" target="_blank">elizabeth.burakowski@unh.edu</a></font></div>
</div>
</div>
</div>
_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.ucar.edu_mailman_listinfo_ncl-2Dtalk&d=DwMFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=JW2rHHzYnVYi6ZDBI-2JFMNSreDPLZO-wwOrczEnOlg&m=dFK61NjWI9bYpf7Dj8JN1OK938G-xKdMzPnKw3vGa_g&s=b0aCHogAHmXT0ALNB7zGXNGY-s_TPZ4Xxxnf9jJN0Mw&e=" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a></blockquote>
</div>
</div>
</div>
</div>
_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a></blockquote></div>