Subject: SUMMARY -- Temperature monitoring [AMD Dual Processor Boxes]

From: "M. Nicklaus" <[email-masked]>
Date: Fri, 25 Apr 2003 16:52:57 -0400 (Eastern Daylight Time)

Dear CCL members,

Quite a while ago I asked the question below; and since John McKelvey's
post from today touches on the same problem, I'm taking this as an
opportunity to summarize the answers I got, as well as to briefly report
on the current status regarding these problems with our cluster.

First off, quite a few people recommended using lm-sensors.  
Unfortunately, this doesn't seem to work well with this specific
motherboard/chipset, so we figured this would not really be of help to us.  
But thanks to everyone suggesting this, in particular to
Greg Galperin <[email-masked]>
Szilveszter Juhos <[email-masked]>
"Christian Muck-Lichtenfeld" <[email-masked]>
Gali Drudis Sole <[email-masked]>

Florian Barth answers proved to be the most interesting ones.  We think it
may indeed be a subtle design flaw of the Tyan Tiger MP dual-Athlon
motherboards that makes them somewhat more heat-sensitive than they should
be, especially for CPU 2.  It appears that some of the components may be
running at the limits of their specifications, and depending on the load,
the cooling setup, the ambient temperature etc., individual motherboards
will show greater or lesser propensity to hang, depending on minor
manufacturing variations.  For some motherboards, these hangs occur at
ambient and CPU temperatures that are quite below what the systems
*should* be able to tolerate.  
    A differential sensitivity between CPU 1 and 2 would also explain some
of the seemingly bizarre behavior we've observed, such as one person's job
dying quite regularly, whereas a different user's jobs virtually never
hanging -- until it became clear that the first user tended to submit jobs
that would typically run in dual-CPU mode, whereas the second user's jobs
were mostly leading to single-CPU runs.

Bottom line: Anything you can do to keep temperature down helps.  We
are now experimenting with stronger fans.  The ones we're trying are
Thermaltake Volcano 7 -- not exactly cheap at $35-$40 apiece.  In
preliminary, very limited tests, they seem to make the system more stable.
No endorsement or final assessment yet, though -- just our first
impression.

Hope this helps,

Marc

##### ORIGINAL QUESTION #####

> We have a Linux cluster consisting of dual XP1900+ Athlon nodes. We've
> found them to be quite heat-sensitive.  We would therefore like to
> monitor component temperature in them (ideally selectively for CPU,
> memory etc.) to pinpoint where the problems lies, and at what
> temperatures the nodes become likely to go down.
>
> What kind of approaches - both hardware and software - have people found
> to be effective for this goal?

##### RECENT POSTING #####

On Thu, 24 Apr 2003 [email-masked] wrote:

> My usual hardware supplier has an outstanding service warranty: bad
> parts pass replacement parts in teh mail for the first year.  They no
> longer like to sell Tyan dual AMD processor systems, but will if the
> customer is willing to deal directly with Tyan when a motherboard
> crashes...  They clain 33% failure rate in the first year.
> 
> I know there are lots of happy AMD dual processor Tyan board users out
> there...  Who has had problems in the last year?

------------------------------------------------------------------------

##### ANSWERS #####

------------------------------

From: Robert J. Doerksen <[email-masked]>

        Not directly answering your question (though I suspect it is the
cpu itself that needs cooling), often the fans in such units are not
sufficient. Try to get them modified so at least the heat will be removed
better from within the boxes. Other than that, air conditioning must be
made strong enough to get rid of the heat that the AMD's produce.

------------------------------

From: Geoff Skillman <[email-masked]>

many of the dual athlon motherboards have built in temperature sensors.

my apologies if you've already checked on this simple solution, but it
would be a pity to miss.  you can often access these sensors through the BIOS
interface.

------------------------------

From: Alessandro Contini <[email-masked]>

Hi, I use a dual MP 1800+ Athlon workstation with a dual SCSI disk array and I
managed in this way:
2 Fans cooling the power supply
2 alloy cooler of big dimension for cpu
2 takimetric 5000 rpm fan for cpu
1 extractor fan
2 fans for the hard disk

With this architecture I can keep the motherboard T < 35C and the cpu's T<
50C.  Obviously for your cluster it's better to keep it in a air conditioned
ambient (max 20C)

------------------------------

From: Forlani Roberto <[email-masked]>

        I haven't tried on our cluster but on my linux laptop I've patched
the kernel for acpi support and it works greatly there are also a lot of
applications (with kde I use akpi) that can visualise graphically the status.

------------------------------

From: [email-masked]

we also have this type of Dual Athlons running. Our general
experience is that outside our air conditioned server room after 3-4
hours (especially in summer) they freeze due to heat problems. In my
opinion no heat monitoring will help to solve this problem. I would
suggest testing different cooling systems like other fan types with
betting silicon grease, Peltier Elements or even liquid cooling.

------------------------------

From: Florian Barth <[email-masked]>

I have build a cluster with MP1800+ processors over the last year. I
also experienced serious instability problems due to temperature and
also thought that the CPUs are the cause. After some (very quick and
dirty) measurements I found out that the CPUs where perfectly inside
their allowed temperature limit (50-65C) but instead some components
(mosfets for the power supply for the CPUs and chipset) of the
motherboard where getting far to hot. Only after cooling them
seperatly with additional fans, I was able to get a stable system. The
boards we are using are Tyan Tiger MP, if you use the same boards,
mabe you have the same problems.

The first thing is, Tyan does not regard to what I
think is a design flaw as a hardware problem they are responsible for
(at least thats the statement from the german technical support). I have
not contacted the Tyan headquaters since I have found a solution how to
run the PCs in the meantime.
  In the cluster we have boards of revision A,B and C (which I think is
the latest). All of them have the same problem that I could fix in the
same way.
  Maybe you have already noticed that the cpu in socket 2 is getting
somewhat hotter than the other cpu. The same problem applies to the
mosfets (transistors) below and left to the socket 2 (there are four of
them if I remember right) which are responsible for the power supply of
cpu2 and parts of the chipset and pci bus.
  These mosfets do fine with Athlons up to 1600+ (a colleague of mine has
some of this boxes) but get far to hot if you use Athlon 1800+ and above
(the board is only specified up to Athlon 1900+). I could not get very
precise measurement, since I lack the equipment to to so (which would be
a heat camera) but when I cool the area below cpu socket 2 (without any
other changes) the whole system runs stable where it failed before
(which is running cpuburn for an hour). Another indication is, that when
you run the system with only one processor at a time and test it with
the cpuburn software it will only fail with cpu 2.
  A Tyan technician insured to me, that the mosfets are still running
inside their specifications but I think they are really at their limits
(some of the boards, about 20%, run fine without additional cooling).
Notably, the MP is the only Tyan dual cpu board not recommended by AMD.
I compared the Tyan MP with its successor, the Tyan MPX. On the MPX they
use different mosfets an far bigger cooling areas (the tin lines on the
back of the motherboard) for them. The MPX is very stable, it runs even
with minimal cooling (just 1 case fan) without failure.

Now to my cooling solution (which is really very selfmade and was quite
work intensive): we disassembled all our PCs (130) and drilled a 60mm
hole in the pc case backplane that holds the motherboard. The hole was
placed right behind the cooling areas for the mosfets of cpu 2. Then we
installed a 60x60x10mm fan in front of the hole and mounted the
motherboard again. Now we have this fan blowing right onto these cooling
areas. I found this solution more reliable than using a fan on top of
the mosfets, which would have been far easier to install. If you want, I
can send you some pictures of the modified case and system.
  Additionally to these fan we have only two more fans in the case (except
for the fan of the PSU): one in the bottom front blowing cool air in and
one in the back blowing hot air out. Both of them are 80x80x25mm fans
with >7000rpm.

------------------------------

From: Morten Stroem <[email-masked]>

We read your posting on the [email-masked] and do believe we have the
software you need to do the necessary monitoring on your system. Our Scali
Manage software have all the features needed to do what you acquire. Please find
attached a pdf file with the data sheet for Scali Manage and our White paper for
the same [omitted -MCN].

Scali Manage Benefits
 *  Single management system for mixed interconnects
 *  Rapid installation/reinstallation of software and nodes
 *  Single point-of-management for one or many clusters
 *  Hardware monitoring for early fault detection
 *  Controlled installation of 3rd party applications & user data
 *  Advanced network and user administration
 *  Remote and secure system architecture
 *  Out-of-band management and disaster recovery
 *  Preventative management and monitoring tools
 *  Scalability to manage small and large systems

##### END OF ANSWERS #####

------------------------------------------------------------------------
 Marc C. Nicklaus, Ph.D.                 NIH/NCI at Frederick
 E-mail: [email-masked]               Bldg 376, Rm 207
 Phone:  (301) 846-5903                  376 Boyles Street
 Fax:    (301) 846-6033                  FREDERICK, MD 21702      USA
          Head, Computer-Aided Drug Design MiniCore Facility
     Laboratory of Medicinal Chemistry, Center for Cancer Research,
 National Cancer Institute at Frederick, National Institutes of Health
       http://rex.nci.nih.gov/RESEARCH/basic/medchem/mcnbio.htm
------------------------------------------------------------------------