[Wrf-users] Nesting and Domain Decomposition

[Douglas Lowe]

Hi all,

I'm running WRF-Chem with a nest of 3 domains, with the settings listed below. I'd like to be
able to split this across as many processes as possible in order to speed things up (currently
I'm only managing 3x real time, which isn't very good when running multiday simulations). 
Unfortunately I am finding that WRF hangs when calling the photolysis driver for my 2nd domain
(which is the smallest of the domains) if I use too many processors.

The (relevant) model domain settings are:
max_dom                         = 3,
e_we                                = 134,  81,   91,
e_sn                                = 146,  81,   91,
e_vert                              = 41,    41,  41,
num_metgrid_levels        = 38,
dx                                    = 15000,3000,1000,
dy                                    = 15000,3000,1000,

WRF will run when I split over upto 168 processes (7 nodes on the ARCHER supercomputer),
but wont work if I split over 192 (or more) processes (8 nodes on ARCHER).

Looking at the log messages I *think* that WRF is splitting each domain into the same
number of patches, and sending one patch from each domain to a single process for 
analysis. However, this means that I am limited by the smallest domain as to how many
patches I can split a domain into before we end up with patches which are dwarved by 
the halos around them.

Would it not make more sense to be able to split each domain into different numbers
of patches (so that each patch is of a similar size, regardless of which domain it is from) and 
send one patch from one domain to a single process (or, perhaps, send more patches from the
outer domains to a single process, if needed for balancing computational demands)? And
is there anyway for me to do this with WRF?

Thanks,
Doug