Subject: Re: CCL:PMEMD 3.1 Release - High Scalability Update to PMEMD

From: Jim Phillips <jim{at}ks.uiuc.edu>
Date: Wed, 5 Nov 2003 20:35:32 -0600 (CST)

Hi again,

On Mon, 3 Nov 2003, Robert Duke wrote:

> reciprocal force computation.  I would like to see a 92K atom benchmark with
> 8 A cutoffs in which you all get 10 nsec/day on 1024 processors at EPCC

Me too.  Stay tuned (but no promises).

> (;-}).  I presumed you all were actually using the larger cutoff to
> compensate for errors associated with the use of a multi-timestep approach

The main limitation on impulse-based multiple timestepping is a non-linear
resonance at half the highest frequency normal mode of the molecule, or
about 5 fs.  This can be higher if you constrain bonds to hydrogens.  If
you're really picky about energy conservation there are other resonances,
but the energy drift is very slight.  One can also use very light Langevin
damping to erase these effects.

> for PME, or simply because using a larger timestep would make the simulation
> scale better, though perhaps have worse low-end performance.  I have never

The apoa1 benchmark started as a cutoff benchmark and had PME added later.
If there is a consensus among the CHARMM and AMBER communities that 8A is
sufficient with PME, then NAMD users can run with 8A.  The effect of
cutoff on NAMD scaling is that there is more work, but there are also
fewer spatial domains to distribute.  For example, apoa1 with 12A cutoff
has only 6x6x4=144 domains, while your FIX (thanks!) at 8A has 11x7x6=462
domains.  These cancel each other out somewhat.

> seen it to be necessary to use such a large cutoff, and indeed pmemd and
> sander 6/7 would have more trouble with really large cutoffs because they
> use pairlists.  We used a 1.5 fs step for both the PME and direct force

NAMD 2.5 uses pairlists, but only 256MB of memory for a 92K atom system
with a 12A cutoff, and the pairlist data is distributed in parallel runs.
You can also turn pairlists off if you're short on memory.

> parts of the calculation, which gives us good results, even with highly
> ionic systems (the FIX benchmark is for a protein with
> gamma-carboxyglutamates and Ca++'s all over the place; the system is from
> published work).

It is a nice benchmark (thanks again).  I hope you can make it public.

> Well, lets do apples and apples on your favorite machine, lemieux, which
> NAMD has been optimized for, and which PMEMD has NOT been optimized for
> (stock mpi; no fancy attempts to structure the code to the machine
> architecture or compiler; in fact I do not know the machine well at all and
> have limited access).  Over the weekend I ran a series of benchmarks on
> lemieux on the JAC benchmark which you referenced below.  I deviated from
> the exact benchmark specification in two inconsequential ways: 1) I did 4000
> step simulations to get more accurate times, and 2) I used a "skinnb" of 1.0
> instead of 2.0.  PMEMD and Sander 6/7 by default check for atom movement
> exceeding the skin, so the skinnb parameter is an optimization parameter,
> and PMEMD does better with it set to 1.0 (ie., this in no way affects
> results of the simulation, PMEMD just builds smaller lists more frequently).

I've just done the same with the NAMD 2.5 released binary.  NAMD 2.5
dynamically adjusts pairlistdist (and temporarily and locally disables
pairlists in the case of a violation), so I started it at 10A (9A cutoff).
This is the time on Lemeiux from the last 1000 steps of a 4000 step run,
using the direct Elan implementation of Charm++ and the "+strategy
USE_MESH" command line option:

 nodes  procs   time (s)

   1      4     222.122
   2      8     115.277
   4     16      59.4164
   8     32      34.3925
  16     64      22.2451
  32    128      12.8739
  64    256       9.90325

I'll leave your data below for comparison.  The first surprise is that for
such different programs (FORTRAN 90 vs C++) the 4 processor times are
shockingly similar, and the race is tied up to 64 processors.  However,
NAMD continues scaling up to 128 processors, at which point we hit the
frustrating 10ms/step barrier.  I've skipped the academic exercise of
running fewer processors per node, and this isn't the right forum to
discuss the fundamental limits of the machine (SC 2003 in two weeks is).
For NAMD the 50% mark is at about 128 CPUs.

> JAC Benchmark, 23558 atoms on lemieux.psc.edu, results in wall clock seconds
> per 1000 steps.
>
> #cpu   #nodes   PMEMD 3.1 wc sec    NAMD 2.4 wc sec   PMEMD/NAMD speedup
>
>     1            1          828.5         1385              1.67x
>     2            1          431.5          750              1.74x
>     4            1          218.25         390              1.79x
>     8            2          118.5          198              1.67x
>   16            4            67.25         105              1.56x
>   32            8            37             61              1.65x
>   64          16            22.25           40              1.80x
>   72          24            19              nd              nd
>   96          32            17.75           nd              nd
> 120          40            18.75            nd              nd
> 128          32            22               23              1.05

In any case, competition is healthy and PMEMD is certainly a worthy and
welcome competitor for NAMD.

Stay tuned to CCL for NAMD FIX benchmark results on the HPCx (EPCC) SP.

-Jim

---

From: "Robert Duke" <rduke)at(email.unc.edu>
Date: Thu, 6 Nov 2003 10:40:35 -0500

Jim -
All I can say is WOW!!!  That is really fantastic scaling on the JAC
benchmark (I have deleted all but the benchmark data in the copy of your
mail below).  I am actually very impressed by two things in NAMD 2.5, based
on this benchmark.  First, there is the clearly superior high end scaling; I
think you all have bested your previous (2.4) high end scaling performance
by something like a factor of 2.  Second, you all have basically also
doubled the performance relative to NAMD 2.4 at the low processor count.  I
had previously had great respect for your scaling, but had considered that
you all were not paying enough attention to performance on small numbers of
processors.  Not any more.  This small numbers of processors performance
issue is really very important; lots of folks out there have to do their
runs on 4-16 cpu's.  So right now, I suspect you all have the scaling title,
and also about equal performance at the low end.  My first reaction to the
scaling was "it must be the elan optimizations on lemieux".  However, on
second thought, more is going on, based on my current understanding of your
architecture.  With your concept of cycles, you manage to make complete
exchange of coordinates occur less frequently (once every 20 steps by
default, I think).  I do complete exchange of coordinates at pairlist
builds, which is typically once every 4 to 7 steps.  I guess I see a clear
direction for PMEMD 3.11.  Stay tuned.  I think this is currently a very
healthy competition, and users of both products will benefit.  I need to
check with Lalith and Lee, but presumably we can make the FIX benchmark more
public.  Congratulations again to the NAMD 2.5 team on a job well done!
Regards - Bob

----- Original Message -----
From: "Jim Phillips" <jim)at(ks.uiuc.edu>
To: "Robert Duke" <rduke)at(email.unc.edu>
Cc: <chemistry)at(ccl.net>
Sent: Wednesday, November 05, 2003 9:35 PM
Subject: CCL:PMEMD 3.1 Release - High Scalability Update to PMEMD

< other stuff deleted>

> -= This is automatically added to each message by the mailing script =-
> To send e-mail to subscribers of CCL put the string CCL: on your Subject:
line
> and send your message to:  CHEMISTRY)at(ccl.net
>
> Send your subscription/unsubscription requests to:
CHEMISTRY-REQUEST)at(ccl.net
> HOME Page: http://www.ccl.net   | Jobs Page: http://www.ccl.net/jobs
>
> If your mail is bouncing from CCL.NET domain send it to the maintainer:
> Jan Labanowski,  jkl)at(ccl.net (read about it on CCL Home Page)
> -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>
>
>
>
>
>