Subject: Re: CCL:file size limits

From: Elmar Gerwalin <[email-masked]>
Date: Tue, 2 Oct 2001 10:30:33 +0200 (MES)

Hi !

On Mon, 1 Oct 2001, Borislav Kovacevic wrote:
> 
> My question is: why are total size of temporary files in Gaussian
(rwf) and in Molpro limited to max. 16 GB under the Linux operating system.

I had a similar question and Gaussian answerded:

-------
 Rev. A.9 on Tru64 Unix uses 64 bit integers for most everything except
it does use a 32 bit hash table.  So in principle you need not split files
but there is a limit, now lifted in Rev. A.10 because we have implemented
a 64 bit hash table,  of about 20GB.  The 16GB limit you refer to is for
machines like Linux where g98 is a 32 bit application.

   You should be able to do this calculation if you specify MaxDisk=20GB
but if you have the disk partitioned you will need to name the RWF as a
set of files with fixed sizes so that they stay below the size of the
partition.

   The SCR file is less used since we have gotten the capability to split
the RWF.  By using only one file we can better utilize the available disk
up to the limit.  If you do use both the SCR and the RWF file you are
still limited and if the SCR file hits 20GB it is done but the disk space
cannot be as easily recovered and re-used as when all of it is allocated
out of the RWF.

   If splitting the RWF and specifying MaxDisk=20GB does not get things
going send the last 50 lines of the output and the first 5 lines, through
the route, of the input.

 Douglas J. Fox
  Technical Support
  Gaussian, Inc.
  [email-masked]

------
Maybe that helps a bit.

Bye
Elmar

============================================================== 
Elmar Gerwalin , University of Kaiserslautern, Germany
                 [email-masked]
                 (soon: ) http://www.rhrk.uni-kl.de/~gerwalin
==============================================================

---

From: "T. Daniel Crawford" <[email-masked]>
Date: Mon, 1 Oct 2001 15:28:09 -0400 (EDT)

On Mon, 1 Oct 2001, Borislav Kovacevic wrote:
>
> Hi CCLers,
> My question is: why are total size of temporary files in Gussian (rwf) and
> in Molpro limited to max. 16 Gb under the Linux operating sistem.
> I know that in 32 bit operating system maximum file size is 2GB.
> In Gaussian I can split rwf files but 8 (16GB) is maximum number of
> them. Why?
> In new version of Linux (kernel 2.4) this 2 Gb problem is solved, i.e. I
> can have a file larger than 2 Gb. I tried with Molpro and sucessfully
> wrote file larger than 2 Gb, but it seems that there is a 16 Gb limit
> anyway.
> Why this limit exists and is there any way to change it?
>
>              Thanks very much!
>

I cannot help with the MolPro limits as I'm not familiar enough with the
details of that code.  However, I can try to clarify a few points regarding
filesize limits under Linux on 32-bit systems (e.g, ix86, athlon, etc.).
This doesn't directly address your question, but does deal with this
general topic and other questions recently posted to this list.

(1) The 2.4.X Linux kernels support filesizes > 2 GB, including under the
ext2 filesystem.

(2) For C-language programs, even with a 2.4.X kernel, one must use the
correct -D flags to make use of large-file support.  These are:

-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE

They must be added for any codes which call low-level I/O routines such
as open(), read(), write(), close(), etc.  This will work for both GNU
and Portland compilers, by the way.

(3) For Fortran programs, if one is using the Portland Group compilers,
PGI has provided large-file support through their I/O library:

http://www.pgroup.com/faq.htm#execute1k

If, instead, one if using the GNU Fortran compiler g77, it is still
possible to obtain code capable of reading/writing large files, but a bit
more problematic.  If anyone cares to do this, let me know and I'll explain
how.  If there's sufficient interest on this list, I'll post it.

-Daniel

-- 
T. Daniel Crawford                           Department of Chemistry
[email-masked]                                    Virginia Tech
www.chem.vt.edu/chem-dept/crawford
                            --------------------
 PGP Public Key at: http://www.chem.vt.edu/chem-dept/crawford/publickey.txt