Subject: Re: CCL:Linux kernel 2.4 parallel scaling

From: Bill Ross <[email-masked]>
Date: Sat, 15 Dec 2001 12:35:12 -0800 (PST)

Myself and others here at NIH have been benchmarking CHARMM and other
	parallel chemistry codes on Linux clusters using MPICH libs with an
	ethernet network.  Tests on 2.4.12 kernel nodes with 2, 4, and 8 procs
	showed terrible parallel scaling, compared to the same tests run with
	the 2.2.16-tcpfix kernel.
	
	...are there some kernel parameters in /proc that can be used
	to tune the communications performance?
	
	Are there any other software solutions to this problem?
	
In another context and programming in java, I noticed that turning 
off Nagle's algorithm on the application side by setting sockets 
to 'nodelay' improved network performance considerably. Watching 
the low-level TCP/IP traffic, we saw that for a send of an 
application-level msg, the protocol went from 4 sequential packets 
with acks to a blast of 4 packets with a single ack coming back. 
When there is any network delay, the sequential mode amplifies it, 
more than I would have expected. 

I'm not sure how this maps to e.g. C - I couldn't find anything
in 'man setsockopt' for example. Looking up "Nagle's algorithm"
& linux on the web,

  http://mosquitonet.stanford.edu/~laik/projects/nagle_test/nagle_test.html

  "This program is useful because some TCP/IP stacks (in particular 
  Linux pre-2.0.30) do not correctly implement Nagle's Algorithm. "

So there is some precedent for problems in this area, and the
other web references in such a search look useful.

I wonder if Nagle's algorithm may be better adapted to network
conditions that are more lossy than usually seen these days,
or for slower computers. If so, I could imagine that it might be
disabled by default in the kernel, at least sometimes, which
could explain the variable results w/ different kernels.

Bill Ross

---

From: David Konerding <[email-masked]>
Date: Sat, 15 Dec 2001 13:15:59 -0800

>other web references in such a search look useful.

Actually, MPICH already disables Nagle's Algorithm, which was designed
to reduce the number of packets going over the wire by agglomeration.  This was
very useful in long-haul links and hub-based TCP/IP from about 5-10 years
ago, but it is somewhat less useful now. 

Also, 2.0.30 is so old as to be completely irrelevant.  Even 2.2 had Nagle
bugs, see:
http://www.icase.edu/coral/LinuxTCP.html

(their site or name server seems to be down).

2.4 should have a much better stack.  But, realistically, programs like
AMBER and CHARMM aren't going to scale very well over TCP/IP due to the
high latency and low bandwidth of 100BaseT (and even gigabit) ethernet.
The best I've been able to get is about 2X for 4 cpus and 3X for 6 cpus.
However, since the cpus are cheap and so is the networking hardware, 
I don't particularly care that the scaling is poor.

Dave