Subject: SUMMARY: Parallel G98 on x86/Linux

From: "M. Nicklaus" <[email-masked]>
Date: Tue, 5 Jan 1999 10:26:54 -0500

A couple of weeks ago, I posted the enclosed question about
running Gaussian 98 in parallel on x86/Linux.  Here's the
summary of the responses I got, slightly edited for brevity.

>From these few replies it appears that no one has been
able so far to get G98 to run properly in this setup.

Maybe Gaussian, Inc. could comment on this.  Doug?

A Belated Happy New Year to everyone.

Marc

------------------------------------------------------------------------
 Marc C. Nicklaus                        National Institutes of Health
 E-mail: [email-masked]               Bldg 37, Rm 5B29
 Phone:  (301) 402-3111                  37 Convent Dr, MSC 4255       
 Fax:    (301) 496-5839                  BETHESDA, MD 20892-4255    USA 
      http://rex.nci.nih.gov/RESEARCH/basic/medchem/mcnbio.htm
    Laboratory of Medicinal Chemistry, National Cancer Institute
------------------------------------------------------------------------

####################### QUESTION #######################

According to Gaussian, Inc. (http://www.gaussian.com/parallel.htm),
G98 is available for Shared Memory Parallel execution on [Intel]
Linux systems.

Does anyone have G98 set up like this, and could report some 
performance results?

I am particulary thinking of a Dual-P II 450MHz system, with
G98 compiled with the Portland Group compilers, and using the
tuned BLAS libraries as recommended by Gaussian, Inc.
(see http://www.gaussian.com/g98_req.htm).  This should be 
about the fastest *inexpensive* G98 implementation on
x86/Linux at this time.  (One can certainly get higher
perfomance by using Linda with >2 CPU's, and/or Xeon processors,
but either option would increase the cost substantially.)

If someone has this setup and could run some benchmarks for me
(should be < 1 h), I'd appreciate.

######################## ANSWERS ########################

From: Drake Diedrich <[email-masked]>

   If the SMP mode uses standard System V IPC it might be possible to
substitute Distributed IPC (http://wallybox.cei.net/dipc) and run Gaussian
in parallel on an arbitrarily large cluster, not just an SMP.  I'd have to
know something about Gaussian's parallelization strategy to even guess
whether it would be efficient though.

-------------------------------------------------------------

From: [email-masked]

I am trying to compile G98 on such a PentiumII 450MHz dual processor .
At present I am not been able to succesfully compile the code with the
blas-opt libraries tuned for dual processors ( ldblasppro1.1m_11.98.a).
It seems that an unsatisfied reference to pthread_create module aborts
the linking procedure. I think that either the libpthread.a library is 
not searched for or G98 does not support parallel execution in that machine. 
This last point has been suggested to me by Doug Fox , but I do not understand 
it well. Do you have any suggestion to solve the problem ?

On the other hand I have compiled G98 with and without the single
processor blas optimized libraries. I have performed both HF and DFT
single point calculations and obtained similar CPU times with both
executables(!). The %Nproc keyword seems not to work with these executables.
Of course, once I solve the dual processor problem I could perform the 
benchmarking job you asked for.

-------------------------------------------------------------

From: JENSAB <[email-masked]>

G98 does not execute in parallel on Intel/Linux systems in our hands.
(See the following email to Dr Dough Fox, Gaussian presumably are 
working on it)

Single processor executions however shows quite satisfactory 
cpu preformance (though Gaussian must have some 32 bit 
windows95/98/NT compiler). I am not sure both machines were running 
100 MHz bus speeds, which should affect the IO intensive gaussian jobs.

To:               [email-masked] (Doug Fox)
Subject:          Slow g98 on Linux/PGIf77 unable to %NPROC=2
Date sent:        Fri, 30 Oct 1998 15:57:30 +0100

Dear Dr. Doug Fox,

Our Linux/PortlandGroup F77 compiled Gaussian98 cannot run in
parallel. Using a single processor it is somewhat slower than the
corresponding G98W running under windowsNT!!?! Here are the Job cpu
times (in sec.) for single processors execution on dual pentiumII 266
MHz machines running the DFT test Job from M. C. Nicklaus, R. W.
Williams, B. Bienfait, M . Hodocek, J. Chem. Inf. Comput. Sci. Vol
38(5) p 893-905 (1998):

# B3LYP/631G(d) and subsequently # B3LYP/631G(d) GEOM=CHECK GUESS=READ

Job cpu / sec.                  
g98W/NT   8074   1975   
g98/Linux/PGI   8594   2101     + ~ 6 %
g94/Linux/f2c   15300   3845    + ~ 91 %

We do not observe any difference in Job cpu between using:

BLAS = blas.a

and 

BLAS = /usr/local/g98/lib/blas-opt.a /usr/local/g98/lib/blas-f2c.a

lsblaspent1.1d_06.98.a (Pentium) and lsblasppro1.1h_08.98.a
(PentiumPro) yield similar results, while ldblasppro1.1h_08.98.a (dual
PentiumPro) gives compilation errors.

The tests were run with no load on the second cpu, where we have
observed a quite severe cpu time penalty when running two identical
single processor jobs simultaneously (~ 30 - 40 % increase in
Gaussian98 reported "Job cpu" times, which are in close to perfect
agreement with "wall clock" cpu time). This is why we are very
interested in the performance of the parallel version.

Given g98 inability to utilize two processors and the reported
performance, why are we bothering with the UNIX version?? Are we the
only ones having problems with g98 or are reports pouring in?

We are however impressed with the g98W which performs really nicely,
and THIS program now does compare favorably with traditional 64 bit
high-end UNIX workstations and presents a very attractive choice.

Yours Sincerely Jens Abildgaard

The following extracts from 3 different NPROC=2 tests with different
MEM settings (0.5MW, 4.19MW and 4.2MW) leave us, as you may by now
expect, completely in the dark. (the route sections for testb and
testc are identical to testa)

test a: MEM=0.5MW

********************************************
 Gaussian 98:  x86-Linux-G98RevA.3 2-Sep-1998
                 28-Oct-1998 
 ********************************************
 %NPROC=2
 Will use up to    2 processors via shared memory.
 %MEM=500000
 ----------------
 #P B3PW91/STO-3G
 ----------------
 1/38=1/1;
 2/17=6,18=5/2;
 3/11=2,25=1,30=1/1,2,3;
 4//1;
 5/5=2,32=1,38=4,42=-6/2;
 6/7=2,8=2,9=2,10=2,28=1/1;
 99/5=1,9=1/99;
 Leave Link    1 at Wed Oct 28 19:35:38 1998, MaxMem=     500000 cpu: 
      0.3 (Enter /usr/local/g98/l101.exe)
 ---------------------------------------------------------------------
 - Geometri Optimeret i B3PW91/6-311G(d,p) Enol uden intramolekylaeaer
 hyd rogenbindi
 ---------------------------------------------------------------------
 - Symbolic Z-matrix: Charge =  0 Multiplicity = 1 C C                
    1    R1 C                    2    R2       1    V2 C              
      3    R3       2    V3       1    D3       0 C                   
 4    R4       3    V4       2    D4       0 O                    2  
 R5       1    V5       3    D5       0 S                    4    R6  
     3    V6       2    D6       0 H                    6    R7      
 2    V7       1    D7       0 H                    3    R8       2  
 V8       1    D8       0 H                    1    R9       2    V9  
     3    D9       0 H                    1    R9       2    V9      
 3    -D9      0 H                    1    R11      2    V11      3  
 D11      0 H                    5    R12      4    V12      3    D12 
     0 H                    5    R12      4    V12      3    -D12    
 0 H                    5    R14      4    V14      3    D14      0
       Variables:
  R1                    1.50489                  
  R2                    1.36216                  
  R3                    1.43593                  
  R4                    1.5135                   
  R5                    1.33334                  
  R6                    1.65523                  
  R7                    0.96216                  
  R8                    1.08624                  
  R9                    1.09284                  
  R11                   1.09402                  
  R12                   1.0929                   
  R14                   1.09347                  
  V2                  120.67031                  
  V3                  130.03436                  
  V4                  114.89229                  
  V5                  117.33213                  
  V6                  126.32056                  
  V7                  109.49169                  
  V8                  113.95904                  
  V9                  110.80924                  
  V11                 111.7517                   
  V12                 109.68088                  
  V14                 112.9822                   
  D3                  180.                       
  D4                  180.                       
  D5                  180.                       
  D6                    0.                       
  D7                    0.                       
  D8                    0.                       
  D9                   59.77432                  
  D11                 180.                       
  D12                 121.55924                  
  D14                   0.                       
 Leave Link  101 at Wed Oct 28 19:35:39 1998, MaxMem=     500000 cpu: 
      0.6 (Enter /usr/local/g98/l202.exe)
 .
 .
 .
 .
 .
(Enter /usr/local/g98/l502.exe)
 Warning!  Cutoffs for single-point calculations used.
 IExCor= 408 DFT=T Ex=B+HF Corr=PW91 ScaHFX= 0.2000
 ScaDFX=  0.8000  0.7200  1.0000  0.8100
 IRadAn=      0 IRanWt=     -1 IRanGd=            0 ICorTp=0
 Using DIIS extrapolation.
 Closed shell SCF:
 Requested convergence on RMS density matrix=1.00D-04 within  64
 cycles. Requested convergence on MAX density matrix=1.00D-02.
 Requested convergence on             energy=5.00D-05. Integral
 symmetry usage will be decided dynamically. IEnd=     11941 IEndB=   
  11941 NGot=    500000 MDV=    494540 LenX=    494540 Fock matrices
 will be formed incrementally for  20 cycles.

 Cycle   1  Pass 1  IDiag 1:
 Petite list used in FoFDir.
 MinBra= 0 MaxBra= 1 Meth= 1.
 IRaf=       0 NMat=   1 IRICut=       1 DoRegI=T DoRafI=F ISym2E= 1
 JSym2E=1. PrsmSu:  requested number of processors reduced to:   1
 ShMem   1 Linda. E=-0.100792091983499D+04 DIIS: error= 3.65D-01 at
 cycle   1. T= 1004. NK=0  NO(<0.9)=   0  NV(>0.1)=   0  31.00e < EF
 0.00e >EF  Err=8.7D-11 RMSDP=2.12D-02 MaxDP=2.07D-01

 Cycle   2  Pass 1  IDiag 1:
 RMSU=  2.07D-02    CP:  9.61D-01
 PrsmSu:  requested number of processors reduced to:   1 ShMem   1
 Linda. Newton Raphson failed in SizSh1. Error termination via Lnk1e
 in /usr/local/g98/l502.exe. Job cpu time:  0 days  0 hours  0 minutes
  0.0 seconds. File lengths (MBytes):  RWF=   10 Int=    0 D2E=    0
 Chk=    1 Scr=    1 Process number  1 failed to complete. Consistency
 failure #1 in CalDSu. Error termination via Lnk1e in
 /usr/local/g98/l502.exe. Job cpu time:  0 days  0 hours  0 minutes
 15.9 seconds. File lengths (MBytes):  RWF=   10 Int=    0 D2E=    0
 Chk=    1 Scr=    1

test b:MEM=4.19MW
(Enter /usr/local/g98/l502.exe)
 Warning!  Cutoffs for single-point calculations used.
 IExCor= 408 DFT=T Ex=B+HF Corr=PW91 ScaHFX= 0.2000
 ScaDFX=  0.8000  0.7200  1.0000  0.8100
 IRadAn=      0 IRanWt=     -1 IRanGd=            0 ICorTp=0
 Using DIIS extrapolation.
 Closed shell SCF:
 Requested convergence on RMS density matrix=1.00D-04 within  64
 cycles. Requested convergence on MAX density matrix=1.00D-02.
 Requested convergence on             energy=5.00D-05. Integral
 symmetry usage will be decided dynamically. Keep R1 and R2 integrals
 in memory in canonical form, NReq=     1685467. IEnd=     11941
 IEndB=     11941 NGot=   4190000 MDV=   2911014 LenX=   2911014
 Symmetry not used in FoFDir. MinBra= 0 MaxBra= 1 Meth= 1. IRaf=     
 0 NMat=   1 IRICut=       1 DoRegI=T DoRafI=F ISym2E= 0 JSym2E=0.

 Cycle   1  Pass 1  IDiag 1:
 Newton Raphson failed in SizSh1.
 Error termination via Lnk1e in /usr/local/g98/l502.exe.
 Job cpu time:  0 days  0 hours  0 minutes  0.0 seconds.
 File lengths (MBytes):  RWF=   10 Int=    0 D2E=    0 Chk=    1 Scr= 
   1 Process number  1 failed to complete. Consistency failure #1 in
 CalDSu. Error termination via Lnk1e in /usr/local/g98/l502.exe. Job
 cpu time:  0 days  0 hours  0 minutes 17.6 seconds. File lengths
 (MBytes):  RWF=   10 Int=    0 D2E=    0 Chk=    1 Scr=    1

test c: MEM=4.2MW
shmget failed.: Invalid argument

---

From: [email-masked] (Mike Frisch)
Date: Tue, 5 Jan 1999 13:07:05 -0500 (EST)

> A Belated Happy New Year to everyone.

There are a number of misconceptions in the replies posted with this message.
Let me explain the situation:

1.  The Linux version can use SMP parallelism for HF and DFT using the usual
    system-V shared memory and unix fork/join calls.  This does not require
    the parallel blas library.  It does require the following:

    a.  A machine with more than one procesor and Linux compiled with SMP support,
        and with the kernel parameter for maximum shared memory segment set
        to some large value (the maximum amount of memory you'll want to give
        G98, typically 3/4 or more of the amount of physical memory on the machine).

    b.  An input deck for a reasonably large job (i.e., not water STO-3G!)
	with %nproc set to requiest more than one processor.  %nproc works
	by default with the Linux version (even on machines with only 1
	CPU), but the default amount of shared memory allowed by the kernel
	is not sufficent to run anything, so jobs with %nproc will fail
	until the kernel is rebuilt with this limit increased.

2.  We have not attempted to build with the Portland compiler and the parallel
    blas library.  We plan to use the thread features of the Portland compiler
    to make our own parallel calls to the (serial) blas library routines, but
    this is not in the current revision of G98.

3.  Some post-SCF jobs spend most of their time in matrix multiplies, which current
    use the serial blas routines and thus are not yet parallelized.

4.  Running efficiently in parallel in 2 CPUs requires twice as much memory
    bandwidth a running on 1 CPU.  Server vendors like SGI, IBM, and
    Digital/Compaq devote considerable effort and cost to having enough
    bandwidth to keep several processors going.  A system with two Intel
    processors but only enough bandwidth to keep one busy is not going to
    run twice as fast in parallel, nor will it run two separate jobs in the
    same elapsed time as a single job.  The observed performance will depend
    strongly on the details of the hardware.

5.  For networks of Linux machines we do support parallelism in HF and DFT
    calculations using Linda, just as for networks with other models of
    workstations.  Linda is required only for network parallelism, not for SMP
    paralleism.

6.  The use of the tuned blas library produces a substantial (serial)
    performance improvement in those jobs which spend a lot of time doing
    matrix multiplies.  It doesn't make much difference for jobs which spend
    their time doing integrals.  So it will make a big difference for QCISD(T)
    but not for HF.

7.  We use the same compiler (Portand) and blas libraries on both Windows/NT
    and Linux, and performance is similar.  The differences in reported times
    are a few percent and probably reflect Linux reporting system time
    overhead (as other unix versions do) while this isn't included on a
    per-process basis under NT.  Elapsed times are sometimes a bit better
    under Linux because of better I/O, but the best summary of the situation
    is that there isn't a signficant performance diffence for running a
    single, stand-alone job under one OS or the other.

Mike Frisch