Subject: Re: CCL:EIDE vs SCSI & backup in Linux clusters

From: Szilveszter Juhos <[email-masked]>
Date: Thu, 6 Jun 2002 07:31:47 +0000 (GMT)

> current perspectives on disk solutions for Intel based Linux clusters?  

IMHO it is better to have a central RAID NFS server and good network
connections than managing individual HDs on the cluster. (You do not need
a HD at all on a cluster node though in practice most people buy nodes
with HD since it is cheaper or just because that is easily available from
vendors. Booting can be solved using network cards with EPROM). For RAID
arrays I would recommend SCSI with a good controller and fine tuned kernel
(summary here:
http://server.ccl.net/cgi-bin/ccl/message.cgi?2002+05+07+009 ). Than
backup becames easier since you have to backup only the RAID array not the
individual nodes.

Hope it helps:
Szilva

---

From: Rick Venable <[email-masked]>
Date: Thu, 6 Jun 2002 18:05:54 -0400

> backup becames easier since you have to backup only the RAID array not the

However, for large scratch files such as one might need for QM
calculations, a local disk would be preferred (faster access) over an
NFS mounted RAID server, which has network overhead.  I do agree that
one doesn't want to use the cluster nodes for any other type of storage.

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
Rick Venable           29/500
FDA/CBER/OVRR Biophysics Lab
1401 Rockville Pike    HFM-419
Rockville, MD  20852-1448  U.S.A.
(301) 496-1905   [email-masked]
ALT email:  [email-masked]
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=

---

From: Malcolm Gillies <[email-masked]>
Date: Thu, 6 Jun 2002 10:21:15 +0200

On Wed, Jun 05, 2002 at 04:37:02PM -0400, Peterson,Brian K. wrote:
> We are specifying a Linux Cluster to use for a variety of codes including
> computational chemistry- both quantum and molecular simulations.  I've
> looked at the CCL archives to see what people have said about SCSI vs EIDE
> drives in such a cluster and the results are a bit ambiguous.  What are the
> current perspectives on disk solutions for Intel based Linux clusters?  Are
> EIDE drives sufficient?  Does one need to tune parameters to get good
> performance?  

SCSI vs IDE can turn into a bit of religious issue, so you are likely
to run into some conflicting opinions. The technology is also changing
quickly, with various different generations of interface available
on the market simultaneoulsy, which makes matters more confusing.

The general opinion seems to be that with the more recent generations
of IDE interfaces (UDMA 66 and up), IDE no longer has a systematic
transfer rate disadvantage compared to SCSI, i.e. for both SCSI and IDE,
interface transfer rates are well above the maximum physical transfer
rates of the available drives.
 
However, new drive technology usually appears first with a SCSI
interface. For instance, I believe that 15 000 rpm drives are currently
only available as SCSI. If you need the highest possible performance,
then SCSI is the only option. On the other hand, SCSI drives tend to be
larger, noisier and dissipate more heat.

The remaining advantage of SCSI for me is the ability to swap drives
easily, and to connect drives in an external enclosure. As far as I
know, hot-swapping of IDE drives is currently impossible.  While this
is useful for file servers, or when configuring machines, it's less
important for scratch disks on cluster machines.

We have a cluster here with both SCSI-based and IDE-based nodes. If
we buy new nodes in the future, I would recommend IDE, as the price
advantage is quite convincing. If disk performance is a limiting
factor, a RAID setup may also be solution.

As for tuning parameters, this will depend on your hardware. For most
chipsets, with a recent Linux kernel (2.4.*) it is no longer necessary.
Ideally, you should test before you buy. With Intel x86-based hardware,
it should be easy enough to find an alternative if a particular type of
motherboard/chipset turns out to be problematic. The rule of thumb
here is to stick with hardware that has been on the market for
a while, say six months to a year.

> Second question.  What are you doing about backup of disks on your Linux
> clusters?  Do you just move output files to another machine which is backed
> up?

We don't back up anything on the compute nodes. The queuing system
(OpenPBS) is responsible for moving output files back to the master
node, which is backed up.

cheers,

Malcolm
--
Malcolm Gillies <[email-masked]>
Post-doctoral researcher
Organic Chemistry, Dept Chemistry,
Technical University of Denmark

---

From: Gregory Shamov <[email-masked]>
Date: Thu, 6 Jun 2002 22:05:14 +0400

Hello Szilveszter,

>> current perspectives on disk solutions for Intel based Linux clusters?  

SJ> IMHO it is better to have a central RAID NFS server and good
SJ> network connections than managing individual HDs on the cluster.

For common parallel qchemistry packages (say, GAMESS) for diskless
cluster you will need _very_ good connections for multigigabyte NFS
disk I/O. I'm not sure that Ethernet (even Gigabit Ethernet) will be
fast enough.

SJ> (You do not need  a HD at all on a cluster node though in practice
SJ> most people buy nodes  with HD since it is cheaper or just because
SJ> that is easily available from vendors. Booting can be solved using
SJ> network cards with EPROM). For RAID  arrays I would recommend SCSI
SJ> with a good controller and fine tuned kernel (summary here:
SJ> http://server.ccl.net/cgi-bin/ccl/message.cgi?2002+05+07+009 ).
SJ> Than  backup becames easier since you have to backup only the RAID
SJ> array not the  individual nodes.

-- 
Best regards,
 Gregory                            mailto:[email-masked]