Subject: Direct versus conventional SCF - summary of responses

From: "Moshe Olshansky" <[email-masked]>
Date: Fri, 1 Sep 95 18:13:29 IST

Dear netters!
I am getting many requests to send a summary of responses to my question
about direct versus conventional SCF,  so here it is:

The original question was:

Dear netters!
About a month ago someone mentioned that conventional SCF is faster (the total
wallclock time) than the direct one only for systems with 100 - 150 basis
functions.  This looks strange to me - isn't it true that for a large number
of ERIs both the time needed to compute them or to read them from the disk is
almost a linear function of their number,  so one method should be always
faster than the other (as soon as the number of basis functions is large enough)?
Could that person please explain his/her claim (either to CCL or directly to me)?

Below are the responses (stripped off headers and signatures):

=========================================================================
From: Jiri Czernek <[email-masked]>
Date: Thu, 24 Aug 1995 18:56:46 +0200 (MET DST)

A thorough discussion of the topic is in "Exploring Chemistry with
Electronic Structure Methods", mainly Exercise 2.5: Resource Usage.
If you use GAUSSIAN you most probably have this book.

I think the topic is the graphs of functions you mentioned can
INTERSECT - there is a crossover point (it depends on the computer
system; e.g., in the exercise from "Exploring ..." it was around
60 basis functions) and "those" total CPU time diferences result
(depending on whether you have lower - conventional SCF faster -
resp. higher - direct faster - number of basis functions than is
the value in a crossover point).

The another topic is neither function you mentioned does not have
the same slope on all its domain.

=========================================================================
From: [email-masked] (Martin Kaupp)
Date: Thu, 24 Aug 1995 19:01:15 +0200 (MES)

I guess the idea was (I'm not the original ''claimer'') that below 100 functions
you can keep the integrals in core memory on typical machines (which is
of course the fastest of all possibilities), whereas above
150 functions the direct algorithm is faster than the conventional.

=========================================================================
Date: Thu, 24 Aug 95 12:10:08 CDT
From: [email-masked] (Ron Shepard)

I wasn't the original poster, but I can give you the standard argument.
In Direct SCF, the small ERIs can be screened at three levels: atom
quartet, shell quartet, and density.  Early in the SCF procedure, loose
thresholds may be used in the first two levels to eliminate the effort
associated with unimportant integrals; near convergence the thresholds
may be tightened to ensure sufficient accuracy.  But near convergence,
the density doesn't change very much, so the last level of screening
becomes effective (F is computed in this case as F=Fold+F(DeltaD)).  With
conventional SCF (I/O based), it is usually not possible to avoid the
I/O on small integrals, so this kind of screening is not beneficial.

There is some controversy in the literature about how effective such
screening can be.  For the DeltaD screening, some claim an order of
magnitude saving of effort, while others claim that it is only marginally
better than simply computing a new F straight from D.

BTW, no one claims to be able to compute integerals faster than doing I/O
on them.  That would be nice, but so far it is not possible.  The Direct
SCF argument always involves these screening/tolerance/threshold type
arguments that benefit more Direct SCF than I/O-based SCF.  The screening
affects more large molecule cases than small molecule cases, and it works
best for large linear molecules, then for large planer systems, and
finally for large space-filling molecules.

=========================================================================
Date: Thu, 24 Aug 1995 14:52:04 -0230 (NDT)
From: Uli Salzner <[email-masked]>

Dear Moshe,
 that direct SCF is faster than conventional SCF for 100-150 basis
functions is a rule of thumb. Which method performs better depends first of all
on your computer. For direct SCF the time to evaluate integrals is crucial, for
conventiuonal SCF it is crucial how fast input output operations can be done.
Conventional SCF also needs to find the right integrals, that is it has to sort
through huge numbers of intergrals. Timings for input/output operations,
searching for integrals and calculation of integrals do not necessarily increase
parallel. The direct procedure is especially competitive since after calculating
the integrals once, efficient cutoffs can be used. That is small integrals will
not be recalculated but neglected.

=========================================================================
Date: Thu, 24 Aug 1995 20:12:23 +0100 (BST)
From: A J Turner <[email-masked]>

This ties in with what I am doing right now.  What you say is resonably
correct except that disk access does decrees in speed with file size
because of the number of separate times the unix (if your using unix - I
cannot comment on VMS etc) subsystem has to read the disk to locate the
end bits of large files.  If the disk is badly fragmented - this gets
even worse.

Thus, I would expect that in some situations, especially on heavily
loaded disks, with multiple users (therefore trashing the cash) there
could be a point where direct gets faster than disk scf.

=========================================================================
Date: Thu, 24 Aug 1995 16:10:23 -0700 (MST)
From: [email-masked] (Xianzhang Gu)

I read your post about conventional vs direct SCF. Although I am not the
person who made that claim, several years ago I implemented direct SCF
method with Gaussian86 which I ported completely on PC-DOS. Here I try
to give my comments about speed comparison.
>From point of view of CPU time, conventional SCF is always faster than
direct one. But for clock time, it is quite different. When using PC or
single user machine, waiting or seeking time (harddrive) can be very
long especially when the harddisk is near full or the file consists of
small separate peices on harddisk ( which means you spend more time to try
to find those peices). If machine got multiple users or many jobs running
concurrently, you have to wait to get your turn for using disk resources.
Generally there are some limit on how big the disk space each user can
use or say each job can use to avoid that one job may eat up all the
time of disk resources. If you have big enough diskspace, I think,
conventional SCF would be faster. The problem is diskspace running out
quite quickly.
Secondly, CPU time needed to compute 2-e integrals is proportional to
ns**n (n is around 3 something) and n decreases while ns ( number of basis )
increases(of course, very slowly). The time needed for SCF part is also
proportional to ns**m ( m is around 2.6, it depends on the method how
you form Fork matrix). m does not change much while ns increases compared
to n ( m may increase a little bit). This is because certain method is used
to screen 2-e integrals before they are actually calculated. So computation
of SCF part is getting equally or more critical when ns gets bigger.
Hope this may give you some hints about speed comparison.

=========================================================================
Date: Fri, 25 Aug 1995 09:57:43 +0200 (MET DST)
From: Patrick Bultinck <[email-masked]>

Well, I once posted a similar question to CCL. From the discussion that
followed and own experience (ab initio with upto 400 BF's) I can tell the
following :

Direct SCF is in my experience always slower, i.e. needs more CPU time;
and more wall clock time.
That is if you have some more or less modern storage, I can imagine that
if you were to put everything on tape (...) you would be faster with
Direct, but otherwise, no. Modern disks (SCSI II f/w) are so fast that
you have to lose whan you have to recalculate your integrals every time.

When you do have an advantage (and even shorter wall clock times) is when
you're working on an I/O slow machine, let's say your average university
central computer with 250 people reading their mail and 200 more doing
LaTex...). So if you have such a machine, it is *sometimes* better to do
direct.

Other reasons include :

- program cannot pack the number of integrals (e.g. GAMESS : max 360 BF's
can be packed into an integral file)
- You don't have 6 Gigabytes Scratch...
- From my experience I hace found that *sometimes* when you're using a
pvm or tcgmsg job over several UNIX boxes you're faster off with direct,
but that's up to you to find out at your site.

Maybe the most remarkable thing about a direct SCF is the following :
when you do a GAMESS run; you will find that
CPU time is about 75 % of real time for conventional SCF
CPU time is about 99 % of real time for direct SCF. Maybe you can
extrapolate what CPU you need to make direct become faster than
conventional... (you have ibm.com, so...)

Anyway, you are not restricted to conventional or direct SCF, you may
also use what is called the Multiplicative Integral Approximation (MIA)
developped at the University of Antwerp...

********************************************************************************
                    THE END
********************************************************************************

Once again,  I wish to thank all those who responded.
Moshe Olshansky
[email-masked]