[ncl-talk] Is there a way to run NCL on multiple processors?

[jbuzan]

Yes, I agree.  I recommend using shell scripts that feed NCL the specifics.  For example, I have scripts that will submit 100s of ncl routines that are executed on individual processors on a cluster.  This aides you in a couple of ways: 1) you have short NCL scripts, which reduce programming time, 2) they are less likely to have errors script errors, and 3) you speed up your throughput dramatically.

-Jonathan


On Oct 13, 2014, at 2:22 PM, Colin Zarzycki <zarzycki at umich.edu> wrote:

> It's worth pointing out that for some applications, you can write shell scripts which can pseudo-parallelize NCL by running multiple instances (which are in serial) alongside one another.
> 
> Ex: If you have a high-resolution file which contains 20 time records and you would like to use NCL to calculate vorticity from U/V for each of the 20 time records, you could spawn 20 instances of NCL which each operate on 1 of the time records. Then concatenate the output in serial after each of the 20 jobs finishes. This will be much faster than reading the entire dataset into a single instance of NCL.
> 
> I have found this technique highly useful when processing very large datasets (or large numbers of files).
> 
> -Colin
> 
> On Sun, Oct 12, 2014 at 9:29 PM, 杨玉华 <yangyh at mail.typhoon.gov.cn> wrote:
> Dear Kyle Griffin,
> Thanks for your great information.
> 
> Yuhua
> 
>  
>  
> 您于 Sun, 12 Oct 2014 22:03:49 -0500 的来信中写到: 
> 
> There is the ParVis project, based on being able to run a select few (but crucial) functions from NCL on multiple processors:
> 
> http://trac.mcs.anl.gov/projects/parvis/wiki
> 
> Unfortunately, the only binaries provided are for system types not widely used in the atmospheric science community (as far as I'm aware) and the build instructions aren't particularly detailed. As such, I've never gotten the ncl included in the ParVis project to work. There is a separate mailing list for the discussion, should you have any problems and/or success with the software. Beyond this, though, I'm not aware of any other efforts to parallelize NCL.
> 
> 
> Kyle
> 
> ----------------------------------------
> Kyle S. Griffin
> Department of Atmospheric and Oceanic Sciences
> University of Wisconsin - Madison
> Room 1421
> 1225 W Dayton St, Madison, WI 53706
> Email: ksgriffin2 at wisc.edu
> 
> 2014-10-12 21:56 GMT-05:00 杨玉华 <yangyh at mail.typhoon.gov.cn>:
> Hi all,
> Is there a way to run NCL on multiple processors?
> Thanks,
> Yuhua
> 
> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ 
> 名称： 中国气象局上海台风研究所 
> 地址： 中国上海市徐汇区蒲西路166号 
> 电话：（＋86）021-64391966 
> 传真：（＋86）021-64391966 
> ============================
> 
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> 
> 
> 
> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝ 
> 名称： 中国气象局上海台风研究所 
> 地址： 中国上海市徐汇区蒲西路166号 
> 电话：（＋86）021-64391966 
> 传真：（＋86）021-64391966 
> ============================
> 
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> 
> -- 
> Colin M. Zarzycki, ASP postdoctoral fellow
> Atmospheric Modeling and Predictability
> National Center for Atmospheric Research
> http://www.colinzarzycki.com
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20141013/87c39d5b/attachment.html