Subject: Summary: Spartan and LoadLeveler?

From: [email-masked] (Ingo Steller)
Date: Wed, 27 Sep 1995 12:33:10 +0100 (MET)

Dear CCLer's:

	Last week I posted a question about possibilities to submit 
SPARTAN jobs into LoadLeveler queues. Many thanks to people who replied 
to this request. Appended below are the responces. Additionally, I want 
to give a short summary.

	The answers can be divided into two groups. First, many people 
suggestet to use the command line interface of spartan (spartan -x job) 
to put the jobs into the queue. That's the easiest, but not the most 
comfortable way, bacause you cannot submit the job without leaving the 
interface.
	Secondly, some people wrote own scripts or changed the 
SPARTAN-scripts to do this job. Among these Farid Parpia from IBM works 
on the problem and he told me, that he will have the tested scripts at 
the end of this week (29 September) (see below). I think he will offer 
the scripts in CCL too.

	Regards,

	Ingo.

Original message:

        I am looking for a possibility to submit Spartan-jobs from the
interface to a queuing system like IBM's LoadLeveler. Does anyone did the
job to write appropiate scripts?  

If not, I will start working on it in the near future.  

I'll summarize all responces in the usual way.  

Sincerely, 
Ingo Steller

-----------------------------------------------------------------------------

Date: Wed, 20 Sep 1995 16:44:19 +0200
To: [email-masked] (Ingo Steller)
From: [email-masked] (Per-Ola Norrby)
Subject: Re: CCL:Spartan and LoadLeveler?

        Dear Ingo,

        I've written a queue system, to which I can submit Spartan,
Gaussian, and MacroModel jobs.  It's single user as of now, but can very
easily be modified for multi-user by specifying a common batch directory.
You can specify slow (-s) and fast (-f) queues as options.  You submit by
writing "po_sp" within the molecule directory or "po_sp molname" one level
above.  There are probably some bugs there still...  Please tell me if you
want the other files (for Gaussian94 and MacroModel) also.  Put the
following two files somewhere on the path:

po_batch:
********************************************************************
#!/bin/sh
#

log=_$$

rundir=${$}d

anydone=0

if [ ! -d $rundir ]
then
    mkdir $rundir
fi

for q in .fast .queue .slow
do

    if [ -f $q ]
    then

        if [ -f _po$q ]
        then
            if [ $anydone -eq 1 ]
            then
                echo "Queue $q already running, quitting">>$log
            fi
            break
        fi

        touch _po$q

        job=""
        lines=`cat $q|wc -l`

        until [ $lines -le 0 ]
        do

            until [ -f "$job" ]
            do

                lines=`cat $q|wc -l`
                if [ $lines -le 0 ]
                then
                    break 2
                fi

                mv $q ${q}_old
                curr=`sed -e 1q ${q}_old`
                sed -n -e 1\!p ${q}_old >> $q
                rm ${q}_old

                job=$curr.job

            done

            echo ">>>Queue: " $q >> $log
            echo " " $job Opened >> $log
            echo $curr > _po$q
            mv $job $rundir/$$.job
            cd $rundir
            csh $$.job >> ../$log
            anydone=1
            rm $$.job
            cd ..

        done

        rm _po$q

    fi

done

rm -r $rundir

if [ $anydone -eq 1 ]
then
    echo "Batch Ended" >> $log
fi
********************************************************************
po_sp:
********************************************************************
#!/bin/csh
#
# This script submits a Spartan job to the batch queue driven by
# po_batch.  If the daemon is not running, it is started.
#

set batchdir=~/batch
if (! -d $batchdir) mkdir $batchdir

if ( $#argv == 0 ) then
    set argv=$cwd:r
endif

set q=".queue"
set donice="nice +10"
set dorun=po_batch
set endmail=0

while ( $#argv > 0 )

    cd $cwd

    set jobroot=$1:r

    shift

    if ( "$jobroot" =~ -? ) then

        switch ( $jobroot )
        case -q:
            set q=".queue"
            set donice="nice +10"
        breaksw
        case -f:
            set q=".fast"
            set donice="nice +0"
        breaksw
        case -s:
            set q=".slow"
            set donice="nice +19"
        breaksw
        case -n:
            set dorun=""
        breaksw
        case -m:
            set endmail=1
        breaksw
        endsw

        if ( $#argv == 0 ) then
            set argv=$cwd:r
        endif

        continue

    endif

    if ( -d $jobroot ) cd $jobroot

    set jobhome=`pwd`

    set jobroot=$jobroot:t

    # Check for an input file

    if ( ! -f _spartan ) then
        echo "$jobroot, no input found"
        continue
    endif

    if ( -f _inuse ) then
        echo "$jobroot, already running"
        continue
    endif

    set count=0
    set xfile=$jobroot
    while ( -f $batchdir/$xfile.job )
        set count=`expr $count + 1`
        set xfile=$jobroot\_$count
    end

    set final

    if ( $endmail == 1 ) then
        set final="echo $xfile complete | mail $user"
    endif

    cd $batchdir

    cat <<END >$xfile.job
    echo " " $xfile Started
    echo " " Home directory $jobhome
    cp -r $jobhome .
    touch $jobhome/_inuse
    $donice spartan -x $jobroot
    cd $jobroot
    rm -f g92* core
    mv * $jobhome
    cd ..
    rm -r -f $jobroot
    rm -f $jobhome/_inuse
    echo " " $xfile Complete
    $final
END

    echo $xfile >> $q

end

$dorun &
********************************************************************

        Hope you can use it,

        Per-Ola Norrby

 *  Per-Ola Norrby
 *  The Royal Danish School of Pharmacy, Dept. of Med. Chem.
 *  Universitetsparken 2, DK 2100 Copenhagen, Denmark
 *  tel. +45-35376777-506, +45-35370850    fax +45-35372209
 *  Internet: [email-masked], [email-masked]

-----------------------------------------------------------------------------

From: "Joe Leonard" <[email-masked]>
Date: Wed, 20 Sep 1995 07:55:04 -0700
Reply-To: [email-masked]
To: [email-masked] (Ingo Steller)
Subject: Re: CCL:Spartan and LoadLeveler?

Ingo,

	I can imagine two ways of doing this (and folks might have already done
this):

1. Modify the files sp_local and sp_remote (which actually run the job steps)
to submit to LoadLeveller rather than run the Spartan binaries directly.

2. Create an external-to-Spartan shell script which takes a molecule (or a
molecule list) and runs it while working with LoadLeveller.

Either way, email your changes to us or the Spartan mail exploder to let others
use it.  If you want to know more about how Spartan's shell scripts interact,
check the file .../spartan4.0/help/Notes/Spartan_Organization.

Joe

-- 

------------------------------------------------------------------------
Joe Leonard Wavefunction Inc. 18401 Von Karman, Suite 370 Irvine, CA 92715
I am a professional... 714-955-2120 do not attempt this at home.
714-955-2118 fax [email-masked]

-----------------------------------------------------------------------------

Date: Wed, 20 Sep 1995 17:34:55 +0200
From: Tim Perkins <[email-masked]>
To: " (Ingo Steller)" <[email-masked]>
Subject: Re: CCL:Spartan and LoadLeveler?

Ingo,

Can I suggest DQS as a very good (free) queuing and load balancing
system.  If your jobs can be writen as (c)sh scripts, then they can
probably be submitted to DQS.

You can get DQS from ftp.scri.fsu.edu in /pub/dqs

The downside is that it takes some time to understand, install, and
correctly figure the software.  However, if you have many
compute-intensive jobs spread over a number of workstations, it's a
very effective solution.

Tim -- Dr Tim Perkins, Drug Design Group Internet:
[email-masked] Department of Pharmacology Phone:  +44 1223
334023 University of Cambridge Fax:  +44 1223 334040 Tennis Court Road
CAMBRIDGE UK CB2 1QJ

-----------------------------------------------------------------------------

Date: Wed, 20 Sep 1995 18:21:22 +0200 (MST)
From: "T. Koch" <[email-masked]>
To: Ingo Steller <[email-masked]>
Subject: Re: CCL:Spartan and LoadLeveler?

Hallo!
Versuchs mal unter 
http://www.chemie.fu-berlin.de/chemnet/use/spartan.html, da steht ein 
bisserl was.
Viel Spass!

Thorsten

/-----------------------------------------------------------------\
|                        Thorsten Koch | | Institut fuer Physikalische
Chemie II der Universitaet zu Koeln | | Luxemburger Str. 116, 50939 Koeln,
Germany | | [email-masked] | | Tel. +49 [0]221 470 4816 |
\-----------------------------------------------------------------/

-----------------------------------------------------------------------------

Date: Thu, 21 Sep 1995 09:45:55 +0800
From: Tsu-chien Weng <[email-masked]>
To: [email-masked], [email-masked]
Subject: Re:  CCL:Spartan and LoadLeveler?

Dear Ingo:
    I have no idea about IBM's LoadLevel, but I have tried to submit
my jobs in SGI one by one. My method is to write down a script. The 
content of the file is as following

spartan -x jobname1
spartan -x jobname2
 ......
    Now, go back shell and change its mode by typing "chmod u+x 
scriptname". Execute the script and it will submit those files
one by one.
    Maybe I misunderstand your words. Hope it helpful.

sincerely,

David Weng

Date: Thu, 21 Sep 1995 02:23:12 -0400
From: [email-masked] (Joe M Leonard)
To: [email-masked]
Subject: CCL:submitting jobs to Spartan...

Folks, this isn't really appropos to the list, but I wanted
to "correct" David Weng's post re: Spartan jobs.  With our latest
stuff, you can also run a bunch of molecules by copying them
all into one directory, marking it as a list by

	touch _spartandir

and then submitting the lot to Spartan via

	spartan -x list_dir_name

this will runs the individual frames in the list via the scheduler,
which can include multiple frames on a multi-processor machine, or
multiple frames on multiple machines via basket scheduling.  This
might be easier than mucking about with shells, particularly
in heterogeneous/networked setups w/multiple Spartan nodes.

For Spartan-specific stuff, you should follow this up off CCL,
but I'm sure many packages have added more-complex job scheduling
logic over the years...

Thanx for reading...

Joe Leonard
[email-masked]

Date: Thu, 21 Sep 1995 09:00:57 -0400
From: [email-masked] (Farid Parpia)
To: [email-masked]
Subject: Submitting jobs to LoadLeveler from the Spartan 4.0 builder

Ingo:

I am working on the above problem, and expect to have complete, tested
scripts for the purpose by the end of next week (29 September).

Regards,

            Farid A. Parpia
            POWERparallel Systems
            IBM Corporation
            Poughkeepsie, NY 12601

From: [email-masked] (Michael Shephard)
Subject: Re: CCL:Spartan and LoadLeveler?
To: [email-masked] (Ingo Steller)
Date: Fri, 22 Sep 95 14:46:26 EDT

I have modified the spartan job submission scripts 'spartand' and
'sp_local' to send the spartan jobs to an NQS queuing system.

The spartand script required the most modification, while the
sp_local script only needed slight modification (merely to delete
all commands related to the Spartan supplied simple queueing system,
as those functions are now taken over by NQS)

The modified spartand script does the pre-processing, the queue selection
procedure, and writes a job-submission script (which is created on the
fly for each job submitted). This job submission script is sent to
the queueing system and contains the commands to run the spartan job and
the post-processing functions of the original spartand.

As the spartan package comes with some utility programs to determine
the type of calculation (to ensure the correct compute module is run)
I used these to automate the queue selection procedure (as our queues
are based on job type, rather than job size or priority).

NB: These spartand and sp_local scripts only work for submissions to
the local machine. For remote submission of Spartan jobs, you may
have to further modify these scripts (if remote submitting via an NQS pipeline
for example) or, if using the Spartan-supplied remote job submission
option, you'll have to modify the sp_remote  script.

Also, these modified scripts have not been tested on multiple-molecule
type jobs

Hope you find these of some use.

Michael

Modified spartand script
====================================================
#!/bin/sh
#
# Modified to use NQS by Michael Shephard 6/7/95
#
# spartand for SPARTAN version 4.0 - modified to use sp_local for actual job
# execution.  This higher-level routine is suitable for queue systems, as well
# as multiple-file execution (as in a Spartan molecule directory).
#
# Spartan defaults to /usr/spartan being present - to find remote spartan's.
# R_INIT_V31 contains /usr/spartan, and so is backwards compatible with
# earlier versions.
#
    R_INIT="/var/spartan/spartanrc"
    export R_INIT

    R_INIT_V31="/usr/spartan/spartanrc"
    export R_INIT_V31
#
# In V4.0, the calling parameters have been modified - the following
# combinations are allowed:
#
#	<jobname>
#	-host <hostname>
#	-user <username>
#
    R_HOST="@@"
    R_USER="@@"
    JOBNAME="@@"

    while [ $# -gt 0 ]
    do
	case $1 in
	-host)
	    shift
	    R_HOST=$1
	    shift

	    ;;
	-user)
	    shift
	    R_USER=$1
	    R_HOST="${R_USER}@${R_HOST}"
	    shift

	    ;;
	*)
	    JOBNAME=$1
	    shift
            ;;
        esac
    done

    export R_HOST
    export R_USER
    export JOBNAME

    if [ $JOBNAME = "@@" ]
    then
	echo 'Job name not defined'
    fi

#
# Things have also been changed to use sp_sched - a means of running multiple
# frames on multiprocesors, and changed again to handle the new list-based
# operations.
#
# Check that there's a $HOME/.spartanrc and $HOME/.spartanrc/submit_spartan
#
    if [ ! -d $HOME/.spartanrc ]
    then
	mkdir $HOME/.spartanrc
	touch $HOME/.spartanrc/submit_spartan
    fi

    if [ ! -f $HOME/.spartanrc/submit_spartan ]
    then
	touch $HOME/.spartanrc/submit_spartan
    fi
#
# Check if we're sitting in a spartan directory.  If so, stay here.  If not,
# cd to $JOBNAME if it's a directory...
#
    if [ ! -f _spartan ] && [ ! -f _spartandir ]
    then
	if [ -d $JOBNAME ]
	then
	    cd $JOBNAME
	fi
    fi
#
# Modified to support both the V3.1 and V4.0 file structure
#
    . $SPAR_SUBMIT/sp_setfnames
#  
# If it's a conformation analysis using ONLY the new logic, continue in 
# spartand with the C_HOST variable set.  Otherwise, spin off sp_net as before.
#
    C_HOST="@@"; export C_HOST

    if [ $R_HOST != "@@" ]
    then
        C_HOST=$R_HOST; export C_HOST

	if [ ! -r $INPUT ] || [ -r _spartandir ]
	then
	    exec $SPAR_SUBMIT/sp_net		# no $INPUT or not a list
	fi

        $SPAR_SUBMIT/sp_choose <$INPUT
        run_stat=$?

        if [ $run_stat -ne 5 ]
        then
            exec $SPAR_SUBMIT/sp_net		# not a CONFANAL
        fi

        $SPAR_CONFORMER/sp_consel $JOBNAME > /dev/null
        run_stat=$?

        if [ $run_stat -eq 1 ] || [ $run_stat -eq 2 ]
        then
            exec $SPAR_SUBMIT/sp_net		# Osawa or Monte Carlo
        fi
    fi
#
# Post that the job has been initiated (although might still be queue'd)
#
    $SPAR_SUBMIT/sp_postjob $JOBNAME $$ -pending
#
# Set the pid for all future calls of sp_postjob
#
    APPROPRIATE_PID=$$
    export APPROPRIATE_PID
#
# Check whether it's a single- or multiple-molecule job.  If multiple-
# molecule job, check that list.ctl is present.
#
    ch_stat=0
    if [ -r list.ctl ]
    then
	$SPAR_SUBMIT/sp_choose <list.ctl
	ch_stat=$?
    fi

    if [ -f _spartandir ] && [ ! -r list.ctl ]
    then
	mult_job=YES
	any_errors=NO

    elif [ -f _spartandir ] && [ -r list.ctl ] && [ $ch_stat -ne 13 ]
    then
	mult_job=YES
	any_errors=NO

    else
	mult_job=NO
    fi

#
# Begin addition to allow use with NQS
#

#Remember the job directory so we can return to it from the NQS spool
#directory
cdr=`pwd`

# Mail notification off by default (set to "-me" to have it default to on)
me=""

# Assign values (actual NQS queue names) to
# the "potential" NQS queues

# Semiemphirical
    qsemiemp=semiemp
# Sybyl Mol Mechanics
    qsybyl=semiemp
# MM2 and MM3 Mol Mechanics (incl Allinger's)
    qmm=semiemp
# Ab initio (incl Gaussian)
    qabinitio=abinit
# Conformational Analysis
    qconform=semiemp
# Mulliken analyis
    qmulliken=semiemp
# DFT calc
    qdft=abinit
# Superposition module
    qsuperposit=semiemp
# MSA
    qmsa=semiemp
# Coordinate driving
    qdyncon=semiemp

#
# Determine the Queue the job is to goto by determining the job type
# NB There probably not as many queues as
# the "nqs_q" variable is assigned to, they
# are just here in case a new queue is created and
# will ease sending relevant jobs to that queue
#

if [ -r list.ctl ] && [ $ch_stat -eq 13 ]
    then
	$SPAR_SUBMIT/sp_choose true < list.ctl
	choose_stat=$?
else
	$SPAR_SUBMIT/sp_choose true < $INPUT
	choose_stat=$?
fi

case $choose_stat in
	1)
		# Semiemphirical
		nqs_q=$qsemiemp
		;;	
	2)
		# Sybyl Mol Mechanics
		nqs_q=$qsybyl
		;;
	3)
		# MM2 and MM3 Mol Mechanics
		nqs_q=$qmm
		;;
	4)
		# Ab initio
		nqs_q=$qabinitio
		me="-me" # Turn email notification on
		;;
	5)
		# Conformational Analysis
		nqs_q=$qconform
		;;
	6)
		# Submit to Gaussian
		nqs_q=$qabinitio
		me="-me" # Turn email notification on
		;;
	7)
		# Submit to Allinger MM3 program
		nqs_q=$qmm
		;;
	8)	
		# Torsions only Mol Mechanics minimisation
		nqs_q=$qmm
		;;
	9)
		#Mulliken analyis
		nqs_q=$qmulliken
		;;
	10)
		# DFT calc
		nqs_q=$qdft
		me="-me" # Turn email notification on
		;;
	11)
		# Superposition module
		nqs_q=$qsuperposit
		;;
	12)
		# MSA thingy
		nqs_q=$qmsa
		;;
	14)
		#Coordinate driving
		nqs_q=$qdyncon
		;;
	*)	
		echo " ERROR: Can't determine job type sorry." >> $OUT
		exit 98
		;;	
	esac

#
# Create the NQS jobs submission script
#

cat <<END >$JOBNAME.job
#!/bin/sh

#Back to originating directory
cd $cdr

#
# Most of the rest of this script is the remaining section of spartand
# now "wrapped" in a 'here' file so it becomes the script file submitted to
# NQS and hence is only fired up when it reaches the head of the queue.
#
# An important change to this segment of spartand is escaping the return
# values of the programs (variable $?) and a couple of other variables
# to avoid them being substituted at this script creation stage. 
#

#	
#End of addition to allow use with NQS
#

    if [ $mult_job = YES ]
    then
	files=\`ls\`

	DOING_LIST=YES
	export DOING_LIST
	$SPAR_SUBMIT/sp_sched local $files
	mult_stat=\$?

	if [ \$mult_stat -ne 0 ]
	then
	    any_errors=YES
	fi

    else
	DOING_LIST=NO
	export DOING_LIST
	$SPAR_SUBMIT/sp_sched local $JOBNAME
	run_stat=\$?
    fi

    if [ $mult_job = NO ]
    then
	case \$run_stat in
	0)
	    errstr="COMPLETED"
	    ;;

	2)
	    errstr="Optimization ran out of cycles "
	    ;;

	14)
	    errstr="Terminated by user (at end of cycle) "
	    ;;

	50)
	    errstr="Spartan/G92 interface error "
	    ;;

	51)
	    errstr="MM3 (Allinger) not installed "	
	    ;;

	52)
	    errstr="Spartan/MM3 interface error "
	    ;;

	58)
	    errstr="Mulliken not installed "
	    ;;

	59)
	    errstr="Error occured while using Mulliken "
	    ;;

	65)
	    errstr="Lists cannot be created within lists "
	    ;;

	88)
	    errstr="Properties module error "
	    ;;

	89)
	    errstr="Graphics module error "
	    ;;

	92)
	    errstr="Gaussian 92 not installed "
	    ;;

	97)
	    errstr="DGEOM not installed "
	    ;;

	98)
	    errstr="Input file not found "
	    ;;

	*)
	    errstr="Check output file "
	    ;;
	esac

    else
	if [ \$any_errors = YES ]
	then
	    errstr="Check output file "
	    run_stat=1

	else
	    errstr="COMPLETED"
	    run_stat=0
 	fi
    fi
#
# Clean up and go home
#
    /bin/rm -f _inuse

    if [ "\$errstr" = COMPLETED ]
    then
	$SPAR_SUBMIT/sp_postjob $JOBNAME $$ -completed

    else
	$SPAR_SUBMIT/sp_postjob $JOBNAME $$ -error "\$errstr"
    fi

    exit \$run_stat

#
# Begin addition to allow use with NQS
#
END

#
# Now submit it.
#
#QSUB options are:
#       -q $nqs_q - Submit to queue $nqs_q
#       -eo - direct stderr output to the stdout destination
#       $me - If set to "-me" this will send mail to job owner on completion
#       -nr - declare that batch request is not restartable
#       -r  - assign stated request name to the request
#       -x  - export all environment variables with request
#       -o $1.log - direct stdout output to the stated destination
#       $1.job - script to execute.
qsub -q $nqs_q -nr $me -eo -r $JOBNAME -x -o $JOBNAME.log $JOBNAME.job >> $JOBNAME.log
rm -f $JOBNAME.job

#
# End addition to allow use with NQS
#
====================================================

Modified sp_local script
====================================================
#!/bin/sh
#
# Modified to use NQS by Michael Shephard  6/7/95
# The mods consist of commenting out all use of the Spartan
# provided queuing software as that function is now taken over by NQS
# and controlled in the spartand script. NQS comment-outs indicated
# by use of ###. If an "if" loop is emptied due to this, a null shell
# command (":") is also added.
# Any nicing of the commands has also been removed.
#

#
# This file is the local-host traffic cop routine.  sp_local assumes that
# it's sitting in the appropriate directory (set by spartand).  Note that
# this requires some extra care when processing a list...
# 
# Notes: The job name's on each execute line in order for the monitor to
#	 pick things up...
#
#	 Conformational analysis is supported in a secondary shell.
#
#	 APPROPRIATE_PID contains the pid required for the sp_postjob call
#	 (the parent spartand).
#
    LOGFILE=sp_err.log
    JOBNAME=$1			# in case of a basket submission

    NO_QUEUE=NO			# set to yes if conf anal w/lists
    DOING_OSAWA=NO		# set to yes if Osawa conf analysis
#
# Check for _spartandir with list.ctl
#
    ch_stat=0
    if [ -r list.ctl ]
    then
	$SPAR_SUBMIT/sp_choose <list.ctl
	ch_stat=$?
    fi

    if [ -f _spartandir ] && [ ! -r list.ctl ]
    then
	if [ -d $JOBNAME ]
	then
	    cd $JOBNAME
	fi

    elif [ -f _spartandir ] && [ -r list.ctl ] && [ $ch_stat -ne 13 ]
    then
	if [ -d $JOBNAME ]
	then
	    cd $JOBNAME
	fi
    fi

    . $SPAR_SUBMIT/sp_setfnames

    if [ ! -r $INPUT ] && [ ! -r list.ctl ]
    then
	exit 98
    fi

    /bin/rm -f $LOGFILE
#
# Check the architecture
#
    $SPAR_SUBMIT/sp_arch
    status=$?

    case $status in
    0)
	USE_ABINIT=$SPAR_ABINITIO
	USE_SEMIEMP=$SPAR_SEMIEMP
	USE_MECHANICS=$SPAR_MECHANICS
	USE_PROPERTY=$SPAR_PROPERTY
	USE_GRAPHICS=$SPAR_GRAPHICS
	USE_SUPER=$SPAR_SUPER
	USE_DFT=$SPAR_DFT

	;;
    *)
	if [ $status -eq 1 ]
	then
	    SUB_DIR=R4K

	elif [ $status -eq 2 ]
	then
	    SUB_DIR=R8K

	elif [ $status -eq 3 ]
	then
	    SUB_DIR=R3K
	fi

	if [ -d $SPAR_ABINITIO/$SUB_DIR ]
	then
	    USE_ABINIT=$SPAR_ABINITIO/$SUB_DIR

	    if [ -r $INPUT ]
	    then
		if $SPAR_SUBMIT/sp_choose parallel <$INPUT
		then
		    if [ -d $SPAR_ABINITIO/$SUB_DIR/parallel ]
		    then
			USE_ABINIT=$SPAR_ABINITIO/$SUB_DIR/parallel
		    fi
		fi
	    fi

	else
	    USE_ABINIT=$SPAR_ABINITIO
	fi

	if [ -d $SPAR_SEMIEMP/$SUB_DIR ]
	then
	    USE_SEMIEMP=$SPAR_SEMIEMP/$SUB_DIR

	else
	    USE_SEMIEMP=$SPAR_SEMIEMP
	fi

	if [ -d $SPAR_MECHANICS/$SUB_DIR ]
	then
	    USE_MECHANICS=$SPAR_MECHANICS/$SUB_DIR

	else
	    USE_MECHANICS=$SPAR_MECHANICS
	fi

	if [ -d $SPAR_PROPERTY/$SUB_DIR ]
	then
	    USE_PROPERTY=$SPAR_PROPERTY/$SUB_DIR

	else
	    USE_PROPERTY=$SPAR_PROPERTY
	fi

	if [ -d $SPAR_GRAPHICS/$SUB_DIR ]
	then
	    USE_GRAPHICS=$SPAR_GRAPHICS/$SUB_DIR

	else
	    USE_GRAPHICS=$SPAR_GRAPHICS
	fi

	if [ -d $SPAR_SUPER/$SUB_DIR ]
	then
	    USE_SUPER=$SPAR_SUPER/$SUB_DIR

	else
	    USE_SUPER=$SPAR_SUPER
	fi

	if [ -d $SPAR_DFT/$SUB_DIR ]
	then
	    USE_DFT=$SPAR_DFT/$SUB_DIR

	else
	    USE_DFT=$SPAR_DFT
	fi

	;;
    esac
#
# Connect to the queue system - users can alter this to conform to
# whatever queueing system is appropriate.  Note: conformational
# analysis must not sit on two slots...
#
    if [ -r $INPUT ]
    then
	$SPAR_SUBMIT/sp_choose <$INPUT
	run_stat=$?

	if [ $run_stat -eq 5 ]
	then
	    $SPAR_CONFORMER/sp_consel $JOBNAME > /dev/null
	    status=$?

	    if [ $status -ne 1 ]	# Osawa[1]
	    then
		NO_QUEUE=YES	# Sequential[0], MC[2], DGEOM[3] or GA[4]

	    else
		DOING_OSAWA=YES
	    fi
	fi
    fi

###    user_name=`whoami`
### 
###     if [ $NO_QUEUE = NO ]
###     then
### 	while [ 1 ]
### 	do
### 	    $SPAR_SUBMIT/sp_queue -start $$ $user_name $JOBNAME
### 	    status=$?
### 
### 	    case $status in
### 	    2)
### 		sleep 15		# this is the "spin-cycle"
### 		;;
### 	    *)
### 		break
### 		;;
###  	    esac
### 	done
###     fi
#
# When here, the job has been released by the queueing system (or was
# never attached to it)
#
    if [ $DOING_LIST = NO ]
    then
	$SPAR_SUBMIT/sp_postjob $JOBNAME $APPROPRIATE_PID -running
    fi
#
# sp_choose looks at the keywords and returns an exit status to indicate
# which Spartan binary is to be used.  This has been changed to support
# a redundent $JOBNAME on the cmd lines for the builder's use.  It has also
# been changed to look at list.ctl first.
#
# DO_FURTHER=1 if Properties and/or Graphics can be run
#
# Various nice settings have been attempted to help preserve interactive
# response
#
    DO_FURTHER=1
    run_stat=0

    if [ ! -r $ARC ]
    then 
	if [ -r $PARC ] 
	then
	    ln -s $PARC fort.11
	fi 

	if [ -r list.ctl ] && [ $ch_stat -eq 13 ]
	then
	    $SPAR_SUBMIT/sp_choose true < list.ctl
	    choose_stat=$?

	else
	    $SPAR_SUBMIT/sp_choose true < $INPUT
	    choose_stat=$?
	fi

	case $choose_stat in
	1)
		$USE_SEMIEMP/sp_semiemp $JOBNAME < $INPUT > $OUT
		run_stat=$?

		/bin/rm -f newarc	# will handle MOLCAS

		;;	
	2)
		$USE_MECHANICS/sp_sybylX $JOBNAME < $INPUT > $OUT
		run_stat=$?
		DO_FURTHER=0

		;;
	3)
		$USE_MECHANICS/sp_mechanics $JOBNAME < $INPUT > $OUT
		run_stat=$?
		DO_FURTHER=0

		;;
	4)
		$USE_ABINIT/sp_abinitio $JOBNAME < $INPUT > $OUT
		run_stat=$?

		/bin/rm -f newarc	# will handle MOLCAS

		;;
	5)
		if [ $DOING_LIST = YES ]
		then
		    echo 'Conformational analysis invalid within list' > $OUT
		    exit 65	# can't make list within list
		fi

		$SPAR_CONFORMER/sp_conform $JOBNAME
		run_stat=$?
		DO_FURTHER=0

		;;
	6)
		$SPAR_G92/translator $JOBNAME > $OUT
		status=$?
		if [ $status -ne 0 ]
		then
		    run_stat=50

		else
		    $SPAR_G92/run_G92 $JOBNAME 
		    stat2=$?
		    if [ $stat2 -ne 0 ] && [ $stat2 -ne 2 ]
		    then
			run_stat=$stat2

		    else
			run_stat=$stat2

			if [ -f $G92 ]
			then
			    $SPAR_G92/loader $JOBNAME >> $OUT
			    stat3=$?
			    if [ $stat3 -ne 0 ]
			    then
				run_stat=50
			    fi
			fi
		    fi

		    if [ -f core ]
		    then
			/bin/rm -f core
		    fi
		fi

		if [ $KEEP_G92_FILES = NO ] || [ $KEEP_G92_FILES = no ]
		then
		    /bin/rm -f *G92*
		fi

		;;
	7)
		printenv MM3_EXEDIR >/dev/null
		status=$?
		if [ $status -ne 0 ]
		then
		    run_stat=51

		else
#
#		    Check whether ssearch is being done
#
		    if grep -i ssearch $INPUT > /dev/null
		    then
			do_conf=1
			method=8

		    else
			do_conf=0
			method=1
		    fi

		    $SPAR_MM3/traverse $JOBNAME > $OUT
		    stat2=$?
		    if [ $stat2 -ne 0 ]
		    then
			run_stat=52

		    else
			/bin/rm -f $MM3 $MM3_MAP
			/bin/rm -f $LIS STOCHASTIC.MEM SORT.LIS

			ln -s $ALI CPD.MM3
			ln -s $MM3_DATDIR/BLANK.DAT PARA.MM3
			ln -s $MM3_DATDIR/CONST.MM3 .

			/bin/rm -f $OUT
			ln -s TAPE4.MM3 $OUT

			$MM3_EXEDIR/mm3 <<% > /dev/null
1
$method
0
%
			/bin/rm -f tmp.F*

			if [ -f TAPE4.MM3 ]
			then
			    /bin/rm -f $OUT
			    mv TAPE4.MM3 $OUT
			fi

			if [ -f TAPE9.MM3 ]
			then
			    mv TAPE9.MM3 $MM3

			else
			    touch $MM3
			fi

			$SPAR_MM3/make_arc $JOBNAME >> $OUT

			/bin/rm -f TAPE4.MM3 TAPE9.MM3
			/bin/rm -f $MM3_MAP
			/bin/rm -f PARA.MM3

			if [ $do_conf -eq 1 ]
			then
			    $SPAR_MM3/get_nbr $JOBNAME
			    num_struct=$?

			    $MM3_EXEDIR/select <<% >> $OUT
$ALI
1
STOCHASTIC.MEM
1 $num_struct
$LIS
%
			    $SPAR_MM3/make_list $JOBNAME $num_struct
			fi

			run_stat=0
			DO_FURTHER=0
		    fi
		fi

		;;
	8)	
		$USE_MECHANICS/sp_rotor $JOBNAME > $OUT
		run_stat=$?
		DO_FURTHER=0

		;;
	9)
		printenv MULLIKEN_ROOT >/dev/null
		status=$?
		if [ $status -ne 0 ]
		then
		    run_stat=58

		else
		    $SPAR_MULLIKEN/pre_mull $JOBNAME > $OUT
		    status=$?

		    if [ $status -ne 0 ] && [ $status -ne 20 ]
		    then
			run_stat=59

		    elif [ $status -ne 20 ]
		    then
##			/bin/rm -f $JOBNAME.run
			$MULLIKEN_ROOT/bin/arcexec $MUL_IN > $OUT

			$MULLIKEN_ROOT/bin/arcexec -a -1 SparExt.in >> $OUT

			if [ -r core ]
			then
			    run_stat=59

			else
			    if [ -r temp.arc ]
			    then
				mv temp.arc $ARC
			    fi
			    run_stat=0
			fi

		    else
			echo 'Mulliken pre-processor created: '$MUL_IN > $OUT
			run_stat=0
		    fi
		fi

		;;
	10)
#
# 999 - this entry must be changed to reflect how DFT's going to be shipped
#
		$USE_DFT/sp_dft $JOBNAME > $OUT
		run_stat=$?

		;;
	11)
		if [ -f $INFO ]
		then
		    mv $INFO save.info
		fi
		
		$USE_SUPER/sp_superposit $JOBNAME > $OUT
		run_stat=$?
		DO_FURTHER=0

		/bin/rm -f */_inuse
		/bin/rm -f .*/_inuse
#
#		Clean up the info files in all sub-dir's
#		(must check V3.1 vs. V4.0 naming scheme)
#
		for i in `ls`
		do
		    if [ -d $i ]
		    then
			cd $i
			if [ -f save.info ]
			then
			    if [ -f input ]
			    then
				/bin/rm -f info
				mv save.info info

			    elif [ -f $i.info ]
			    then
				/bin/rm -f $i.info
				mv save.info $i.info
			    fi
			fi

			cd ..
		    fi
		done

		;;
	12)
		$USE_SUPER/sp_msa $JOBNAME > $OUT
		run_stat=$?
		DO_FURTHER=0

		/bin/rm -f */_inuse

		;;
	14)
		if [ $DOING_LIST = YES ]
		then
		    echo 'Coordinate driving invalid within list' > $OUT
		    exit 65	# can't make list within list
		fi

		$SPAR_DYNCON/sp_dyncon $JOBNAME > $OUT
		run_stat=$?
		DO_FURTHER=0

		;;
	*)	
		echo " ERROR: bad input file format" >> $OUT
		if [ $NO_QUEUE = NO ]
		then
###		    $SPAR_SUBMIT/sp_queue -finish $$ $user_name $JOBNAME
		    :
		fi

		exit 98

		;;	
	esac
# 
# Handle the exit status from the compute modules
#
	case $run_stat in 
	0)			# successfully completed
		if [ -r fort.12 ]
		then
		    mv -f fort.12 $ARC
		fi
		if [ -r fort.11 ]
		then
		    /bin/rm -f fort.11 $PARC
		fi

		;;
	2)			# failed, ran out of cycles	
		if [ -r fort.12 ]
		then
		    mv -f fort.12 $ARC
		fi
		if [ -r fort.11 ]
		then 
			/bin/rm -f fort.11 
		fi

		;;
	14)			# failed, external termination
		if [ -r fort.12 ]
		then
		    mv -f fort.12 $ARC
		fi
		if [ -r fort.11 ]
		then 
			/bin/rm -f fort.11 
		fi

		;;
	51)			# MM3 not installed on target machine

		;;
	51)			# MM3 not installed on target machine

		;;
	58)			# Mulliken not installed on target machine

		;;

	*)			# failed, error not trapped here
		if [ -r fort.11 ]
		then 
			/bin/rm -f fort.11 
		fi

		;;
	esac
    fi

    if [ $run_stat -ne 0 ]
    then
	if [ $NO_QUEUE = NO ] && [ $DOING_OSAWA = NO ]
	then
###	    $SPAR_SUBMIT/sp_queue -finish $$ $user_name $JOBNAME
	    :
	fi

	exit $run_stat
    fi
#
# Only run Properties and/or Graphics if there's QM information in the archive
#
    if [ $DO_FURTHER -eq 0 ]
    then
	if [ $NO_QUEUE = NO ] && [ $DOING_OSAWA = NO ]
	then
###	    $SPAR_SUBMIT/sp_queue -finish $$ $user_name $JOBNAME
	    :
	fi

	exit 0
    fi
#
# Generate propery data
#
    if [ -r $ARC ] && [ -s $PROPIN ]
    then
	ln -s $ARC fort.11
	if [ -r $PROPARC ]
	then
	    ln -s $PROPARC fort.12
	fi
	ln -s $INPUT fort.10

	if grep "SPARTAN DENSITY FUN" $ARC > /dev/null
	then
	    $USE_DFT/sp_dft $JOBNAME >> $OUT
	    prop_stat=$?

	else
	    $USE_PROPERTY/sp_property $JOBNAME < $PROPIN >> $OUT
	    prop_stat=$?
	fi
	/bin/rm -f fort.10

	case $prop_stat in
	0)
	    /bin/rm -f fort.11
	    if [ -r fort.12 ] 
	    then
		if [ -r $PROPARC ]
		then
		    diff fort.12 $PROPARC >/dev/null
		    diff_stat=$?
		    if [ diff_stat -ne 0 ]
		    then
			mv -f fort.12 $PROPARC

		    else
			/bin/rm -f fort.12
		    fi	

		else
		    mv -f fort.12 $PROPARC
		fi
	    fi

	    ;;

	*)
	    if [ -r fort.11 ]
	    then
		/bin/rm -f fort.11
	    fi

	    if [ -r fort.12 ]
	    then
		/bin/rm -f fort.12
	    fi

	    if [ $NO_QUEUE = NO ] && [ $DOING_OSAWA = NO ]
	    then
###		$SPAR_SUBMIT/sp_queue -finish $$ $user_name $JOBNAME
		:
	    fi

	    exit 88

	    ;;
	esac
    fi
#
# Generate graphical data
#
    if [ -r $ARC ] && [ -s $INFO ]
    then
	$USE_GRAPHICS/sp_graphics $JOBNAME >> $OUT
	graf_stat=$?

	if [ -r temp.graph ]
	then
	    mv -f temp.graph $INFO
	fi

	if [ $graf_stat -ne 0 ]
	then
	    if [ $NO_QUEUE = NO ] && [ $DOING_OSAWA = NO ]
	    then
###		$SPAR_SUBMIT/sp_queue -finish $$ $user_name $JOBNAME
		:
	    fi

	    if [ $graf_stat -eq 14 ]
	    then
		ex_stat=14

	    else
		ex_stat=89
	    fi

	    exit $ex_stat
	fi
    fi 
#
# Clean up and return
#
    if [ $NO_QUEUE = NO ] && [ $DOING_OSAWA = NO ]
    then
###	$SPAR_SUBMIT/sp_queue -finish $$ $user_name $JOBNAME
	:
    fi

    exit 0
====================================================

--
_________________________________________________________________________
| Michael Shephard.                  |      [email-masked]      |
| University of New South Wales.     |  [email-masked]  |
| Australia                          |                                  |

-- 
Ingo Steller                            e-mail: [email-masked]
Universitaet-Gesamthochschule Essen     URL: http://www.structchem.uni-essen.de/
Fachbereich 08 - Chemie                 Tel.: ++49 (0)201 183 3599
Institut fuer Anorganische Chemie       Fax : ++49 (0)201 183 2535
Abteilung Roentgenstrukturanalyse

Universitaetsstrasse 3-5
45117 Essen