Subject: Summary: Disk solutions in heterogeneous environments

From: John Shelley <[email-masked]>
Date: Tue, 7 May 2002 10:28:30 -0700

Dear CCL,
   There were 3 very helpful responses to my query about cheap disk-storage 
that can cross mounted on a wide variety of Unix platforms.
   They are listed below (with permission for responses not already posted to 
the list).
   Looks like this can work with some care (kernel problems for which patches
are available seem to be the key).

   I really appreciate the responses provided,

                       John Shelley
                       Senior Scientist
                       Schrodinger, Inc

---------------------

Original Question:

Dear CCL,
    I'm curious about a cheap relatively modest disk solution (100-500
GBytes) that can be effectively cross mounted on a wide variety of unix
systems (Linux, SUN, AIX, HP, Compaq TRU64 and SGI IRIX) using NFS for
scientific data storage/access.  It need not be a RAID system but could
be.

   The main reason that I'm asking about this is that I've heard about
severe problems with access to the disk being frequently frozen with
simultaneously cross mounting relatively cheaps disks in
heterogeneous environments using NFS.

   I'd appreciate any success/problem stories that people might have
regarding this issue.

================================

   Here are the answers received:

================================

From: Jim Phillips <[email-masked]>

We've been using Sun servers running Solaris, mounted by all of those
platforms, stably for years.  (Actually, we just got the AIX box last
week, but everything else has been working solidly for three years.)  You
can get a low-end Sun for $1000.  We've had bad luck with Linux servers.

- - - - - - -
The computer systems used in the Theoretical Biophysics Group, 
Computing Facility an NIH Resource for Macromolecular Modeling and 
Bioinformatics are described at:

http://www.ks.uiuc.edu/Development/Computers/ or
http://www.ks.uiuc.edu/Development/Computers/systems.html

================================

From: Malcolm Gillies <[email-masked]>

Hi John,

I'm not sure I understand what you're looking for, but we have a
Linux NFS server which is mounted by a cluster of 15 Linux machines,
as well as a couple of SGIs. All very simple and cheap, and no problems
since it was installed about 18 months ago. The disks themselves are
SCSI, but I can't see that IDE would make a big difference, apart from
SCSI being easier to swap around (we have fancy rackmount enclosures).

A couple of weeks ago I set up software RAID on a Linux machine for
the scratch partition (hoping to speed up the I/O). That went very
easily using the Red Hat install tools, and so far nothing unusual
has popped up. That's not being served across NFS of course.

What do you mean by "relatively cheap disks"?

With regard to "freezing", I have had two problems with poor response
times across NFS on my Linux machines. The first was simply due to
loading the server with CPU intensive processes (eg jaguar), which
was easily solved by keeping those jobs off the server. The second
was due to a bug in the Ethernet driver on the client machine, which
was causing extremely low network throughput. Fixed with a kernel
upgrade, once I realized what was going on.

- - - - - - -
Follow-up messages:

> > What do you mean by "relatively cheap disks"?

Somewhere in between business class RAIDs with very high reliability and 
disks commonly used on PCs.

> 
> [...]
> the various machines that we need to work on.  We have had some problems
> with large cheap disks and cross mounting them from Linux to a lot of
> other computers.  I've heard that the disk would go into a weird state
> (NFS problem) and that would essentially make it inaccessible on all the
> computers.  Inaccessible is a mild term since doing a df would cause the
> window involved to irreversibly lock up as well.
 
Do you have anything to go on suggesting that the problem was disk-
related rather than purely NFS-related?

(Not really but perhaps kernel related, I was not involved in this.) 

As someone else mentioned, having a dud kernel version can be a problem.
Particularly with older kernels (pre 2.2.14, from memory) the NFS
implementation had some serious problems. Our NFS server is running an
SGI-patched 2.2 series kernel which apparently includes some NFS fixes.
I believe these and many more are included in the current 2.4 series
releases. NFS3 over UDP appears to be the preferred option at the
moment; the TCP implementation is still flaky.

If you want something off the shelf, there are various Linux-based
boxes around, like the Cobalt Qube gear (sold by Sun these days?).
More expensive than rolling your own, of course.

========================================
This one was posted on CCL

From: Szilveszter Juhos <[email-masked]>

For a Linux/BSD based solution go for hardware RAID(5) with a reliable
card ( http://linas.org/linux/raid.html ). You can use NFS automounter
(for Linux sometimes better to use the BSD-based daemon instead of the
kernel automounter). Also NFS 3 client/server is supported for Linux
nowadays. If you are using many inodes/files, it is adviced to increase
/proc/sys/fs/file-max (2.4 kernel and also the inode-max on 2.2. kernels)
to its double/triple value. Also better to use the vm33 patch on a machine
that is used as an application server (though it is always a bad idea to
give the fileserver into the users hand ;) Do not forget that for
automounted volumes you have to explicitly 'cd' into the dir, it is not
enough to use the auto-completion in the shell (the TAB key I mean).

Adviced to use a journalling filesystem BUT NOT reiserfs on such systems.
ext3/xfs is the option, I do not know how IBM's jfs is performing, but
that also can be an option.

Backing up such data is a pain if you do not want to pay for expensive
tape robots, etc. People say amanda (GNU) is OK, I have very good
experiences with arkeia (that is not free for this size unfortunatelly but
still many times cheaper than Veritas or HSM solutions). Also due to the
frequent RPC vulnerabilities better to have at least a packet filter
firewall.

So there is no out-of-the box Linux solution, but with some tuning you can
have a pretty good system. I was managing ~2TB on-line data with similar
architecture in a heterogenous net (Linux,HP,Solaris,Windows,IRIX). The
main source of problems was cr@py hardware RAID cards and out-of-the-box
kernels.