[Wrf-users] wrfv3 & mpi

[Jan Ploski]

Rodrigo Delgado Urzúa wrote:
> 
> Cesar Hernandez Baño wrote:
>> Hi
>>
>> I'm using wrfv3 , with mpich 1.2.7p1. The problem is when I use more 
>> cpus/nodes in the cluster, the more slower runs the test.
>>
>> I have 4 pcs, Pentium 4 , four cpu each server. I run 
>> test/em_real/wrf.exe with mpirun
>>
>> Network cards are Gigabit; using 2 processes network traffic flows at 
>> 70 MBits aprox. Using 12 processes uses 150 MBit.
>> When I use 2 processes, each CPU uses 80-90% aprox. Using 3 processes, 
>> each CPU uses 50-60%.
>>
>> What's the problem?
>>
>> Cheers
>>
> 
> Hi Cesar,
> 
> I think that's a normal thing. The explanation is simple: more processes 
> in different machines implies more communication to perform computing. 
> This happens with relatively slow interconnects (as FastE or GigE) as 
> with the faster ones (e.g., Infiniband), the difference is the fast 
> interconnects have room to allow more traffic and low latencies than the 
> slow ones, so you can add CPUs to the simulation without compromise 
> performance. But even the fastest interconnects can be completely busy 
> when you add several CPUs to the simulation, resulting in lack of 
> performance.

Here's one scalability comparison we did - 03 (OpenMP) vs. 05 (MPI over 
GigE) vs. 19 (MPI over InfiniBand):

https://srvgrid01.offis.uni-oldenburg.de/wrf-benchmark/case.pl?case=conus12km_2001

Up to 8 processes you should get a good speedup. Have you compared 
running multiple MPI processes on a single machine with distributing 
them across machines? Have you tried a different MPI implementation 
(OpenMPI is very easy to compile and install in my experience).

Regards,
Jan Ploski