[Wrf-users] WRF use on large HPC systems

[John Michalakes]

Dear WRF Users,
 
Thank you very much for the kind and very helpful responses to my post the
WRF Users list from earlier this month (original message from Feb 6th,
below).  Here are the results from this informal one-time survey:
 
*         Number of responses: 17
*         WRF Applications (listed alphabetically):
o   Air quality
o   Basic research and model development
o   Regional Climate and Climate Research
o   Hurricane research
o   Operational NWP
o   South Asia Monsoon
o   Severe weather
o   Wind energy
*         Largest number of cores: 140,000 (Hurricane Sandy simulation)
(There was also a response describing an impressive 437K core hurricane
simulation, but here the focus was high-performance computing scalability of
WRF.)
*         Number of applications running on:
o   More than 10,000 cores:  3  (Hurricane, Severe Weather)
o   Between 8,000 and 10,000 cores: 2   (Hurricane, Climate)
o   Between 1,000 and 8,000 cores: 2  (NWP, Basic research -- both were 1024
cores)
o   Between 32 and 1,000 cores:  10   (range of applications)
*         Some of the computer systems used by respondents (ordering is not
significant; apologies if I didn't list your facility):
o   https:// <https://bluewaters.ncsa.illinois.edu/hardware-summary>
bluewaters.ncsa.illinois.edu/hardware-summary
o   https:// <https://portal.tacc.utexas.edu/user-guides/stampede>
portal.tacc.utexas.edu/user-guides/stampede
o   http:// <http://www.idris.fr/ada/> www.idris.fr/ada/
o   http:// <http://www.emsl.pnl.gov/emslweb/capabilities/computing>
www.emsl.pnl.gov/emslweb/capabilities/computing 
o   http:// <http://www.nersc.gov> www.nersc.gov
o   http:// <http://www.dkrz.de/Klimarechner-en/hpc>
www.dkrz.de/Klimarechner-en/hpc 
o   http:// <http://www.bsc.es/marenostrum-support-services/mn3>
www.bsc.es/marenostrum-support-services/mn3
o   http:// <http://www.top500.org/system/177818>
www.top500.org/system/177818
o   http:// <http://rdhpcs.noaa.gov/boulder/> rdhpcs.noaa.gov/boulder/
o   http://www.ccs.tsukuba.ac.jp/eng/
*         Detailed breakdown showing typical number of cores and maximum
numbers used (also attached).
 

 
Typical
Maximum

Hurricane
 
140000

Hurricane
 
46592

Severe weather
 
32000

Hurricane lightning
256
9600

Climate
2000
8188

NWP
896
1024

Research
256
1024

Physics development
300
750

Climate
480
700

Wind energy
 
640

Climate
128
512

Air Quality (CMAQ)
128
256

Hurricane
 
200

S. Asia Monsoon
 
196

Climate
 
192

Climate
32
128

NWP
 
120
 
 
Thanks again for all the responses, and best wishes for continued success in
your research.
 
John 
 
 
 
From: John Michalakes [mailto:john.michalakes at noaa.gov] 
Sent: Friday, February 6, 2015 6:01 AM
To: 'Wrf-users at ucar.edu'
Subject: WRF use on large HPC systems
 
Dear WRF Users,
 
I'm interested in learning about the use of WRF on large high-performance
computing (HPC) systems for research and operations (not large runs for
computational benchmarking or HPC research).  If you're doing WRF
simulations that require large numbers of processor cores (100s? 1000s?
10000s?  More?) for your research, forecasting, or other applications please
contact me and let me know:
 
*         One sentence description of your WRF application
*         Typical number of cores that you use for your WRF runs
*         Largest number of cores that you have used for the WRF runs
*         Where you usually run and a brief description or a URL for the
computing center you use
*         Whether you could use even larger numbers of cores to run WRF if
you had them available?
*         Reason for needing large numbers of cores (speed? problem size?
both? other?)  
 
If you're not sure what the job sizes are in number of "cores", use whatever
unit you're comfortable with (numbers of MPI tasks, number of OpenMP
threads, ...).
 
Thank you!
 
John Michalakes
john.michalakes at noaa.gov <mailto:john.michalakes at noaa.gov> 
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/wrf-users/attachments/20150218/3665e9ca/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: responses.png
Type: image/png
Size: 33116 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/wrf-users/attachments/20150218/3665e9ca/attachment-0001.png