[ncl-talk] Why could Killed happen?
Ian Harris (ENV - Staff)
I.Harris at uea.ac.uk
Fri Jun 14 09:35:43 MDT 2019
Hi Rick,
Thanks for the reply. It’s actually OSX, though in this context I'm using it as a Linux box with extras.
I find the idea that it ran out of memory alarming, as I have 32GB and there was precious little else going on at ten to five this morning. Furthermore, the previous crash was after 16 years (including generation back to 1901) and that was with 16GB. And yep, I do delete all arrays as soon as I’m done with them.
ncl is currently using 25.88GB of memory. I agree that’s a lot, but it’s all available.
I’m processing the files one at a time, as they’re annual with all nine 3-hourly variables in them. All arrays are only as large as they need to be.
If the system is killing the process, it seems to be doing it quietly - I see nothing for times around then in the system logs.
Ah, well, I’ll see if it does it again before it finishes tomorrow.
Cheers
Harry
On 14 Jun 2019, at 15:01, Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>> wrote:
Hi Harry,
A simple "killed" message is Linux's very unhelpful manner of saying it has direly run out out of memory (google that for plenty of references/discussion). It is neither an NCL or a Python issue.
Without better understanding of your script, I'm not sure what to suggest. Perhaps process your files in smaller batches? You are aware that "delete"ing no-longer-used variables frees their memory?
Rick
On Fri, Jun 14, 2019 at 4:11 AM Ian Harris (ENV - Staff) <I.Harris at uea.ac.uk<mailto:I.Harris at uea.ac.uk>> wrote:
Hi,
I’m running a long and slightly cumbersome NCL script - with callouts to WRAPITted Fortran subs. It takes about 8 days to complete, as it’s reading a lot of high res gridded files (HARMONIE/V1 reanalysis), regridding them, and producing an aligned-with-CRU TS forcing dataset 1901-2018. It also sends each file via scp as it’s produced.
However, I’ve now had two instances of it just.. dying. The most recent was this morning (UK time):
(0) Fri 14 Jun 2019 04:47:42 BST: top of monthly loop with iy=2005 and im=8
(0) Fri 14 Jun 2019 04:47:42 BST: top of variable loop with var=TMP_P0_L103_GLC0
cruhar_v3_2005_08_3hourly_Tairalign.nc 100% 162MB 10.8MB/s 00:15
(0) /Volumes/VERIFY2/output_v3/cruhar_v3_2005_08_3hourly_Tairalign.nc successfully uploaded
(0) Fri 14 Jun 2019 04:48:48 BST: top of variable loop with var=PRES_P0_L1_GLC0
cruhar_v3_2005_08_3hourly_Psurfnoalign.nc 100% 162MB 9.5MB/s 00:17
(0) /Volumes/VERIFY2/output_v3/cruhar_v3_2005_08_3hourly_Psurfnoalign.nc successfully uploaded
(0) Fri 14 Jun 2019 04:49:55 BST: top of variable loop with var=WDIR_P0_L103_GLC0
(0) Fri 14 Jun 2019 04:50:39 BST: top of variable loop with var=WIND_P0_L103_GLC0
Killed
[imac:~/WORKING/VERIFY] ianharris%
Now where does that 'Killed' come from? Is the system killing it? I see nothing in the system logs. I have plenty of RAM (double what I had the first time it Killed), so I can’t see it being that.
Any suggestions would be helpful!
Cheers
Harry
PS Yes I’m sure it won’t happen with Python :)
Ian 'Harry'Harris | NCAS-Climate | Climatic Research Unit |School of Environmental Sciences | 01.02 Hubert Lamb Building | Faculty of Science | University of East Anglia | Norwich Research Park | Norwich |NR4 7TJ
Tel: +44 1603 593 818 Fax: +44 1603 591 327
I work 80%FTE, normally Tuesday-Friday
<image001.png>
Gold (Teaching Excellence Framework 2017-2020)
UK Top 15 (The Times/Sunday Times 2019 and Complete University Guide 2019)
World Top 200 (Times Higher Education World University Rankings 2019)
<image002.jpg> <image003.png> <image004.png> <image005.png> <image006.png> <image007.png> <image008.png> <image009.png>
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.ucar.edu%2Fmailman%2Flistinfo%2Fncl-talk&data=02%7C01%7CI.Harris%40uea.ac.uk%7C1d8b2b0422be4eaa45c308d6f0d150b3%7Cc65f8795ba3d43518a070865e5d8f090%7C0%7C0%7C636961179137621678&sdata=Ug1bj4a%2FAre8MUaYl11dH6t0RglN2DCdDbD3OYVc%2B8g%3D&reserved=0>
Ian 'Harry'Harris | NCAS-Climate | Climatic Research Unit |School of Environmental Sciences | 01.02 Hubert Lamb Building | Faculty of Science | University of East Anglia | Norwich Research Park | Norwich |NR4 7TJ
Tel: +44 1603 593 818 Fax: +44 1603 591 327
I work 80%FTE, normally Tuesday-Friday
[cid:image001.png at 01D33EAD.B14304A0]
Gold (Teaching Excellence Framework 2017-2020)
UK Top 15 (The Times/Sunday Times 2019 and Complete University Guide 2019)
World Top 200 (Times Higher Education World University Rankings 2019)
[cid:image002.jpg at 01D33EAD.B14304A0] [cid:image003.png at 01D33EAD.B14304A0] [cid:image004.png at 01D33EAD.B14304A0] [cid:image005.png at 01D33EAD.B14304A0] [cid:image006.png at 01D33EAD.B14304A0] [cid:image007.png at 01D33EAD.B14304A0] [cid:image008.png at 01D33EAD.B14304A0] [cid:image009.png at 01D33EAD.B14304A0]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 6210 bytes
Desc: image001.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 1179 bytes
Desc: image002.jpg
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1726 bytes
Desc: image003.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1799 bytes
Desc: image004.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 1733 bytes
Desc: image005.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 1732 bytes
Desc: image006.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 1784 bytes
Desc: image007.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.png
Type: image/png
Size: 2063 bytes
Desc: image008.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image009.png
Type: image/png
Size: 936 bytes
Desc: image009.png
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190614/9acbac78/attachment-0007.png>
More information about the ncl-talk
mailing list