Subject: Summary: Batch Queuing Systems

From: Bill Laidig <[email-masked]>
Date: Mon, 29 Jan 2001 09:14:16 -0500

Yesterday,  I posted a question about batch queueing systems:

> All,
>
> We have been looking into and testing queuing software for a while and
> found nothing very satisfactory.  All have drawbacks such as price, lack
> of adequate cross-platform support or features.  I am interested in what
> queuing software CCL users USE on their systems and how they like this
> software.  I will summarize responses to the list.  By the way, we are
> looking for a queuing system that:
>
> a) Works across both SGI Irix and Linux clusters (NQS works well on
> linux, but we are having problems getting it work well on SGI's if I
> remember the problems our systems manager mentioned).
> b) Is not extremely expensive (LSF, for example, is quite nice, but
> would be rather expensive to implement across our whole system)
> c) Is full featured in terms of queue control (priority, job limits,
> processor limits, time limits, ability to suspend and track jobs...)
> d) Can treat a group of machines as a single queue (for our cluster)
>

Since then I have received a lot of useful responses.  Thanks to
everyone for their quick, detailed responses.  This is what I learned:
*********************************************************************************

PBS, the portable batch system (http://www.openpbs.org/ or .com or .net
- all work), was recommended by most of the respondents.  It seems to by
flexible, free and work across many platforms.  It also has a scheduling
module that can easily be extended/replaced and several mentioned other,
more flexible schedulers that could be substituted.  This is the system
we will be trying out first.

DQS was also recommended by several list members.  If we have problems
with PBS we will try DQS out.

LSF also received a couple of positive responses and we may reconsider
this if the other choices do not work and we can get the budget.

I also recieved one response favoring generic NQS and one pointing to a
soon to be released free system from Sun, the Grid engine.

Thanks again to everyone and the detailed responses are listed below.

*********************************************************************************

Did you look at PBS?
http://pbs.mrj.com/
http://hpc.sissa.it/pbs/
http://www.chpc.utah.edu/policies/pbs.html

Jan K. Labanowski

*********************************************************************************

Did you look at PBS (Portable Batch System)?

http://www.openpbs.org/

Bill Ross

*********************************************************************************

Try DQS.

However, LSF has become sort of a standard in the commercial world.

-P.

*********************************************************************************

PBS is the answer. It is free/cheap, should work on more than one
machine,
is very flexible in terms of queue control, can treat a group of
machines as a single queue.

The downside of the flexibility is that one has to work on the
scheduler (which is just FIFO by default),
but [email-masked] has done a fine job of writing his own version
(the Fair Share model), which works very well.
He probably can also point you to some more documentation.

Most sincerely,

Marcel Swart.

*********************************************************************************

www.openpbs.net

Groeten, David.

*********************************************************************************

Hi Bill,

PBS is the free queing system of choice.  We run it on SGIs and LINUX
clusters.  NQS is crap, LSF is ok but costs mucho.  PBS, is choice and
you get the source.

-Matt

*********************************************************************************

Bill,
    The following maybe of interest as SUN says that they will be
releasing
free versions for both IRIX and Linux, although they are not ready yet
:-

http://www.sun.com/software/gridware/

Bye Jeff.

*********************************************************************************

On Sun, 28 Jan 2001, Bill Laidig wrote:

> We have been looking into and testing queuing software for a while and

> found nothing very satisfactory.  All have drawbacks such as price,
lack
> of adequate cross-platform support or features.  I am interested in
what
> queuing software CCL users USE on their systems and how they like this

> software.

        We use Open PBS (www.openpbs.com) here and it works well for our

needs (distribution of single-processor and MPI jobs across a network of

Linux-based workstations and compute nodes). It is available under a
fairly liberal licence, and is free for both commercial and
non-commercial
use provided you are registered with its owners. The "P" in PBS stands
for
"portable" and as such it works on a variety of platforms (although
we've
only tried it on Intel and Alpha Linux boxes so far here). It is fairly
feature-rich by default, but since the source code is available, you can

augment its features (or "bugs" if you like ;) with your own. If that's
still not enough, you can get the "full" version, PBS Pro, but that
costs
money...

> a) Works across both SGI Irix and Linux clusters (NQS works well on
> linux, but we are having problems getting it work well on SGI's if I
> remember the problems our systems manager mentioned).

        If you've used NQS in the past, PBS should be relatively easy to

pick up, as it's based (at least in part) on NQS. There are even
utilities
for converting NQS job scripts to PBS format.

> c) Is full featured in terms of queue control (priority, job limits,
> processor limits, time limits, ability to suspend and track jobs...)

        This is all possible by default, I believe, with the exception
of
processor limits (you can restrict jobs by nodes - machines - but not by

processors). Patches are available for this functionality, however (for
instance, from our own site, http://bellatrix.pcl.ox.ac.uk/~ben/pbs/).
PBS
is flexible in that this kind of thing is generally determined by a
dedicated scheduler program, which you can either provide from scratch
(in
one of three different languages) or can derive from the example
"FIFO" scheduler.

> d) Can treat a group of machines as a single queue (for our cluster)

        This is the default behaviour. More complex behaviour can be
obtained by tinkering with the scheduler. Most "unusual" requirements
have
already been addressed by other researchers, however, and you can
usually
find patches that do something similar to what you want on the mailing
list, which is generally quite responsive.

        Ben

*********************************************************************************

Bill,

we are using Generic NQS with GAMESS-US on SGI Indigo2 (Irix 6.5.9),
which is working fine.

Wolfgang

*********************************************************************************

hi bill,

we use DQS on SGI, linux and DEC. There are about 20 machines in our
system, and most of the time it runs stable - well to be honest it can't

remember a breakdown caused by that part of the whole installation ;).
We used NQS before, but when we renewed allmost all our hardware in 1998

it wasn't as easy to learn and showed not enough hardware flexibility to

me. I also remember some comiling problems expecially on SGI.

DQS worked fine after only a few hours of reading, compiling and
installation - and i have only been a beginner on administration of
UNIX-Systems at that time ;).

We still use same version today - after about 2 years running time, I
just did not see any need of upgrading.

let me know if you need some more information or config filed of our
system - good luck,

c.

ps.
oops, did I mention: we use DQS3.2.7 from
http://www.scri.fsu.edu/~pasko/dqs.html

*********************************************************************************

Hi Bill

Try the Distributed Queueing System (DQS). We use it on SGI Irix and
Linux Workstations without problems and it is free software.
Look at www.scri.fsu.edu/~pasko/dqs.html

Greetings from Darmstadt, Germany
Bernd

*********************************************************************************

Bill,

You might remember me as a grad student who worked for Dan Edwards.  We
met
at Sanibel a few years ago. I am now at a pharmaceutical company and
have
been managing a queue for a couple of years that is now made up of
SGI's,
Linux and a new IBM SP. I think you will find that LSF is the product to

meet all your needs and it is not as expensive as first appears when you

consider the increase in efficiency of use of the systems.  I had Eric
Deumens here a few weeks back installing our SP and he has been trying
DQS,
NQE, etc. but was very impressed (I think) with LSF.  I'd be happy to
talk
with you more about this if you want some more info.

Bob

--
************************************************************************
*    "Like jewels in a crown, the precious stones glittered in the     *
*     queen's round metal hat." - Jack Handey                          *
*                                                                      *
*     Bill Laidig                                                      *
*     The Procter & Gamble Co.             tel 513-627-2857 fax - 1233 *
*     Miami Valley Laboratories            [email-masked] (preferred)   *
*     P.O. Box 538707                      [email-masked]            *
*     Cincinnati, OH 45253-8707                                        *
************************************************************************