[ncl-talk] Limiting the number of CPUs when running NCL scripts

[Brian Thomas]

You might try using the “wait” command in the bash script to make it perform the commands sequentially.


Message: 3
Date: Sat, 27 Jun 2020 17:26:56 +0000
From: "Herb, Jason" <jherb at albany.edu<mailto:jherb at albany.edu>>
To: "ncl-talk [ncl-talk]" <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: [ncl-talk] Limiting the number of CPUs when running NCL
       scripts
Message-ID:
       <DM6PR04MB4555FA9AD4425D79FDBCEF8DCE900 at DM6PR04MB4555.namprd04.prod.outlook.com<mailto:DM6PR04MB4555FA9AD4425D79FDBCEF8DCE900 at DM6PR04MB4555.namprd04.prod.outlook.com>>

Content-Type: text/plain; charset="iso-8859-1"

Hello,

I was attempting to run a script that runs ncl_filedump on a series of hdf files and placing each file dump into a seperate file. The script is in a bash environment with the following sample command lines...

#!/bin/bash
#Script will run ncl_filedump for all data days this is the first step to the swath time periods for comparison of terra and aqua
ncl_filedump MCD19A2.A2002001.h12v04.006.2018016140659.pscs_000501413482.hdf > ncldump2002001.txt
ncl_filedump MCD19A2.A2002002.h12v04.006.2018016140659.pscs_000501413482.hdf > ncldump2002002.txt
ncl_filedump MCD19A2.A2002003.h12v04.006.2018016140700.pscs_000501413482.hdf > ncldump2002003.txt
ncl_filedump MCD19A2.A2002004.h12v04.006.2018016140701.pscs_000501413482.hdf > ncldump2002004.txt
ncl_filedump MCD19A2.A2002005.h12v04.006.2018016140702.pscs_000501413482.hdf > ncldump2002005.txt
ncl_filedump MCD19A2.A2002006.h12v04.006.2018016140702.pscs_000501413482.hdf > ncldump2002006.txt
.
.
ncl_filedump MCD19A2.A2002365.h12v04.006.2018018154414.pscs_000501413482.hdf > ncldump2002365.txt

Apparently, the system went and ran all commands at a single time, verses doing in list order. Is there a way to limit the number of CPUs and make it work in the list order to not over load the system? The existing script resulted in the error of unix fork retry no child processes appearing on the screen and locked the system up. I do no know what the exact specs of the computer system are, but i know there is a very large number of CPU cores and RAM.

Thanks,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20200627/17075f95/attachment-0001.html>

------------------------------

Subject: Digest Footer

_______________________________________________
ncl-talk mailing list
ncl-talk at mailman.ucar.edu<mailto:ncl-talk at mailman.ucar.edu>
List instructions, subscriber options, unsubscribe:
https://mailman.ucar.edu/mailman/listinfo/ncl-talk

------------------------------

End of ncl-talk Digest, Vol 199, Issue 34
*****************************************

@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@
Brian C. Thomas, PhD
Professor, Physics and Astronomy, Washburn University

785-670-2144<tel:(785)%20670-2144>
brian.thomas at washburn.edu<mailto:brian.thomas at washburn.edu>

Web: http://www.washburn.edu/faculty/bthomas/
ORCID profile: http://orcid.org/0000-0001-9091-0830
ResearchGate profile: https://www.researchgate.net/profile/Brian_Thomas

Department of Physics and Astronomy
Washburn University
1700 SW College Ave.<https://maps.google.com/?q=1700+SW+College+Ave.%0D+Topeka,+KS+66621&entry=gmail&source=g>
Topeka, KS 66621<https://maps.google.com/?q=1700+SW+College+Ave.%0D+Topeka,+KS+66621&entry=gmail&source=g>

@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20200628/ad07686b/attachment.html>