[cam-users] CGD Forums

[phpbb at cgd.ucar.edu]

Dear cam-users, 

As you requested, here is the latest digest of messages posted on CGD Forums forums. Please come and join the discussion!


<<  Cloud sedimentation blowing up with first sea ice, http://bb.cgd.ucar.edu//viewtopic.php?t=88 >>

PosterpjrPostedTue Jan 18, 2005 5:23 am MST, http://bb.cgd.ucar.edu//viewtopic.php?p=280#280
Message: Hi Matt,
<br />
<br />Did you get past this problem? If not, email me offline or call and
<br />I will try to make some suggestions. This is not one of those things
<br />I can do much about without more info.
<br />
<br />Phil

------------------------------
PostermhuberPostedTue Jan 18, 2005 5:37 am MST, http://bb.cgd.ucar.edu//viewtopic.php?p=281#281
Message: Hi phil,
<br />   Yeah, as became apparent through testing, the ocean was actually getting twitchy and was throwing out some bad points which the ice model was trying to deal with by growing ice, and for some reason these problems culminated in fatal errors in the previously mentioned cam routine.  Not that that's a problem mind you--in this case ccsm should have been bombing since the ocean was becoming numerically unstable.  So the fix was simple, and as pointed out by several helpful people, such as Bruce Briegleb, has been used in some of the higher co2 runs for future conditions.  I decreased the ocean model time step, the ocean stopped throwing bad points, the sea ice went away, and cam stopped bombing.  So the good news is that the simulation is doing ok, the bad news is that this run is really pushing the model to the limits of what it can handle, and my pop time step is getting very small and the model is slow, slow, slow....
<br />
<br />thanks for everyon!
 e's help.
<br />-Matthew Huber
<br />Purdue University
<br />http://web.ics.purdue.edu/~huberm
<br />http://roskilde.eas.purdue.edu/~cdpl/phpwiki
<br />http://www.purdue.edu/eas/people/faculty/huber.html

------------------------------

<<  Why is CAM slower in CCSM3.0 than in CCSM2.0.1?, http://bb.cgd.ucar.edu//viewtopic.php?t=92 >>

PosterworleyphPostedMon Jan 17, 2005 5:15 pm MST, http://bb.cgd.ucar.edu//viewtopic.php?p=278#278
Message: Hi,
<br />    I am the author of much of the current MPI logic in CAM physics and the CAM spectral dycores. 
<br />
<br />1) You can run the code using mpi_sendrecv instead of mpi_alltoallv by adding the following lines to the CAM namelist ...
<br />
<br />dyn_alltoall = 1
<br />phys_alltoall = 1
<br />
<br />2) Unless mpi_alltoallv is seriously broken in your MPI library, I doubt that MPI is the culprit here. Load imbalance shows up in the MPI timers, and the physics cost has increased since CAM2.0. How much, I don't know - you'll have to ask one of the science developers. CAM3 has load balancing options that work in CCSM3, but CAM2 load balancing was never validated in CCSM2 to my knowledge. If you compile CAM (2 or 3) with the the CPP flag TIMING_BARRIERS set (adding -DTIMING_BARRIERS to CPP_FLAGS) you can trap load imbalance in the timing barriers timer events. First, though, you might simply compare min and max timers for the events bc_physics and ac_physics ac!
 ross all processes for both CAM2 and CAM3 to check whether there are obvious differences.
<br />
<br />3) There are LOTS of MPI tuning options in CAM3. However, in my experience, most of them don't make much difference. It might be worthwhile trying load balancing. A low communication option is
<br />
<br />phys_loadbalance = 3
<br />phys_alltoall = 1
<br />
<br />The optimal physics load balance (and maximum communication cost) is
<br />
<br />phys_loadbalance = 2
<br />phys_alltoall = 0 or 1, depending on whether you want to use mpi_alltoallv or mpi_sendrecv to implement the alltoallv
<br />
<br />There are also options to use non blocking send/recv commands if you think that these would be better than mpi_sendrecv. I'd wait until you look at the load imbalance and sendrecv vs. alltoallv issues.
<br />
<br />Hope that this helps.
<br />
<br />Pat Worley

------------------------------
PosterpjrPostedTue Jan 18, 2005 5:16 am MST, http://bb.cgd.ucar.edu//viewtopic.php?p=279#279
Message: I can comment a bit on the physics changes between CAM2 and CAM3.
<br />We have added significantly to the physics cost by putting in explicit transport of two more water species (ice and liquid), sedimentation, and phase changes between condensate species, explicit representation for about 10 species of aerosols, significant improvements in radiative transfer, and cloud overlap, and code modifications that have occasionally sacrificed performance for usability. There is no doubt that CAM3 is more expensive than CAM2.
<br />
<br />
<br />Phil

------------------------------




Format TEXT
Show Message Text YES
Show My Messages YES
Digest Frequency DAY
Show only new messages since last time I logged in FALSE
Send empty digests NO
Time of day to send digest 9 AM
Maximum characters per message in digest 32000