[Wrf-users] Results variability depending on processor count

[Gustafson, William I]

Jan,

This issue is actually a pretty complicated one that is a bit machine and
compiler dependent. In fact, my understanding is that for any of us
developing code to be released with WRF, we must be able to reproduce the
wrfout files and get a bit-for-bit match when we change processor counts.

The reason for the differences are many. The most problematic is bugs in the
code, e.g. an array not being given a value before being used, which leads
to random results. Another possibility is how optimization is done. Each CPU
has a set of registers that hold values used for calculation and
intermediary results. These registers typically operate at a higher
precision than the numbers held in memory. So, when numbers are passed from
a register to memory and brought back to another register, a small amount of
precision is lost. The implication is that if multiple calculations can be
done entirely in the registers, one can gain a little accuracy. However, if
during the same series of calculations one has to use memory space to hold
an intermediary value, the result could differ at the end. Compilers often
have options to prevent these differences by forcing round-off error to be
handled consistently, e.g. with ifort one would add "-fp-model precise".

Hope this helps a bit.

-Bill

___________________________________________
William I. Gustafson Jr., Ph.D.
Scientist
ATMOSPHERIC SCIENCE AND GLOBAL CHANGE DIVISION
 
Pacific Northwest National Laboratory
P.O. 999, MSIN K9-30
Richland, WA  99352
Tel: 509-372-6110
William.Gustafson at pnl.gov
www.pnl.gov/atmospheric/staff/staff_info.asp?staff_num=5716


On 9/19/08 12:13 AM, "Jan Ploski" <Jan.Ploski at offis.de> wrote:

> "James Correia" <jimmyc42 at gmail.com> schrieb am 09/18/2008 04:12:15 PM:
> 
>> Hi-
>> It is more matter of truncation at a certain number of processors
>> specifically involving domain decomposition.
> 
> James,
> 
> Thanks for responding. Can you explain in more detail what you mean by
> "truncation"? After all, the domain decomposition happens at a grid point
> boundary, so in my imagination it should be "clean" without any scraps
> falling off the table. I realize that to compute a grid point's value you
> need all the neighbors, some of which may only be present in another tile,
> but then, it is possible to obtain their exact values by communicating
> with one of the peer processes, so that not a satisfactory explanation
> either.
> 
>> That is using, say 64 processors, you will always get the same answer
>> on a particular machine.
> 
> Yes.
> 
>> But if you use 32 processors, the domain decomposition and
>> subsequent integration produces slight differences
> 
> It is the causal link between decomposition and the differences that I'm
> puzzled about. The after-effects are clear.
> 
> I think the kind of answer I am looking for is a white-box example of 2
> different ways of decomposition on some domain inevitably producing 2
> different results in the subsequent computation. If it is indeed just the
> initial decomposition which matters, then I suppose that already the
> initial integration step would yield different results.
> 
>> I have seen this issue before and even in small time period
>> integrations. The key is the number of processors. I would imagine
>> this is also compiler dependent (i.e. Exactly how functions are
>> calculated and at what precision).
> 
> It could be, but I'm more interested in the case where we hold every
> parameter except the number of MPI processes constant.
> 
> Regards,
> Jan Ploski
> 
> --
> Dipl.-Inform. (FH) Jan Ploski
> OFFIS
> FuE Bereich Energie | R&D Division Energy
> Escherweg 2  - 26121 Oldenburg - Germany
> Phone/Fax: +49 441 9722 - 184 / 202
> E-Mail: Jan.Ploski at offis.de
> URL: http://www.offis.de
> _______________________________________________
> Wrf-users mailing list
> Wrf-users at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/wrf-users