Subject: Summary: F77 compiler and MIPS R10000

From: "Robert Fraczkiewicz" <[email-masked]>
Date: Mon, 25 Nov 1996 13:15:16 -0600

Dear Netters,

Thanks again to all who replied to my posting included below. IMHO, the
discussion that followed it is certainly beneficial to the Computational
Chemistry community, especially those who use Silicon Graphics hardware.

Robert Fraczkiewicz

------------------------------Original posting--------------------------

> Subject: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
> Dear Netters,
>
> Below is a short FORTRAN code that I wrote solely for the purpose of
> testing the system subroutine DTIME in SGI IRIX 6.2. This subroutine
> returns the elapsed CPU time since the last call to DTIME. I have
> compiled it on SGI Indigo2 with R10000 processor using MIPSpro F77
> compiler. I have used five different option sets listed below:
>
> 1) f77 silly.f -o silly        (this produces 32-bit mips2 code by default)
> 2) f77 -O2 silly.f -o silly
> 3) f77 -64 silly.f -o silly    (this produces 64-bit mips4 code)
> 4) f77 -64 -O2 silly.f -o silly
> 5) f77 -64 -O3 silly.f -o silly
>
> The unusual results I have obtained are shown in table below. Apart
> from the fact that the 64-bit executables produce slightly different
> result for x than their 32-bit counterparts, the CPU times in column
> 4) are larger by a factor of TEN in comparison to other columns!
> Normally, I would expect them to be in between 3) and 5). This abnormal
> result is reproducible for every SGI R10000 machine that we have here.
> It seems that options -64 and -O2 don't get along very well. Is there
> something I don't understand about MIPSpro F77 optimization options? Am I
> missing something? Or is there a bug in the compiler?
> Please, recompile this code on your R10000 (if you have it :-) to confirm
> or deny this observation.
>
>       program silly
>       implicit double precision (a-h,o-z)
>       real tarray(2)
>       x=0.d0
>       jmax=100
>       call dtime(tarray)
>       do imax=8000,10000,100
>        do i=1,imax
>         do j=1,jmax
>          x=x+sqrt(float(i))*sin(float(j))
>          y=acos(x/(i*abs(x)))
>         enddo
>        enddo
>        call dtime(tarray)
>        print *,imax,tarray(1)+tarray(2)
>       enddo
>       print *,x,y
>       stop
>       end
>
>
> Results:
> ------------------------------------------------------------------------
>        | CPU times (s) reported by DTIME for different compiler options|
>   imax -----------------------------------------------------------------
>        |     1)    |      2)    |      3)    |      4)    |      5)    |
> ------------------------------------------------------------------------
>   8000 | 0.8353710 |  0.6551640 |  0.6200770 |   6.131035 |  0.1884840 |
>   8100 | 0.8722960 |  0.6899580 |  0.6547980 |   6.208158 |  0.1924730 |
>   8200 | 0.8814020 |  0.6967919 |  0.6354470 |   6.280854 |  0.1928640 |
>   8300 | 0.8918310 |  0.6798510 |  0.6682850 |   6.354552 |  0.1953330 |
>   8400 | 0.9021270 |  0.7131230 |  0.6759790 |   6.454478 |  0.1976650 |
>   8500 | 0.9128000 |  0.7212590 |  0.6588350 |   6.503715 |  0.2249840 |
>   8600 | 0.9232100 |  0.7044880 |  0.6914790 |   6.604474 |  0.2023520 |
>   8700 | 0.9336550 |  0.7377620 |  0.6991270 |   6.682483 |  0.2047420 |
>   8800 | 0.9442470 |  0.7459780 |  0.7070640 |   6.732563 |  0.2070200 |
>   8900 | 0.9545040 |  0.7541710 |  0.6897570 |   6.828230 |  0.2343980 |
>   9000 | 0.9649110 |  0.7372350 |  0.7226390 |   6.902500 |  0.2117620 |
>   9100 | 0.9503450 |  0.7706130 |  0.7303350 |   6.977732 |  0.2140380 |
>   9200 | 0.9860920 |  0.7787640 |  0.7130180 |   7.052145 |  0.2164480 |
>   9300 | 0.9962510 |  0.7868320 |  0.7457520 |   7.127767 |  0.2188510 |
>   9400 |  1.006923 |  0.7699320 |  0.7534280 |   7.226908 |  0.2460940 |
>   9500 |  1.017477 |  0.8030970 |  0.7614270 |   7.281057 |  0.2234870 |
>   9600 |  1.052756 |  0.8112460 |  0.7440030 |   7.351462 |  0.2258430 |
>   9700 |  1.038267 |  0.8196840 |  0.7766720 |   7.452884 |  0.2281600 |
>   9800 |  1.048346 |  0.8278640 |  0.7845440 |   7.500894 |  0.2555520 |
>   9900 |  1.059044 |  0.8359420 |  0.7923810 |   7.600510 |  0.2328600 |
>  10000 |  1.069380 |  0.8191160 |  0.7999970 |   7.650619 |  0.2352500 |
> ------------------------------------------------------------------------
>    x   |   -1522838.034014487   |          -1522834.807409876          |
> ------------------------------------------------------------------------
>    y   |    1.570896326795063   |           1.570896326795063          |
> ------------------------------------------------------------------------
>
> Best regards,
> Robert Fraczkiewicz
> University of Texas Meedical Branch
> Galveston, TX 77555
>
>-- End of excerpt from Robert Fraczkiewicz
============================================================================

The first reply came from Omar Stradella of SGI:

On Nov 21,  7:04pm, Omar G. Stradella wrote:
> Subject: Re: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
> Hi Robert,
>
> That certainly doesn't look right. Did you run all the cases
> standalone ?. You are printing user+sys times, try to print
> both, are the sys times much higher than the user times ?
> I've just ran your program on a "quiet" Indigo2 R10000/195MHz
> and this is what I got:
>
>        --------------------------------------------
>          imax |     3)    |      4)   |      5)   |
>        --------------------------------------------
>          8000 | 0.6389320 | 0.4704070 | 0.3378320
>          8100 | 0.6739330 | 0.4792880 | 0.3438850
>          8200 | 0.6547930 | 0.5068420 | 0.3709340
 ..........................................................
>
> As you said, -O2 should always be faster than -O0 (the default).
> -O3 is usually faster than -O2, but not always it depends on the
> code.
>
> Omar.
>
>-- End of excerpt from Omar G. Stradella
========================================================================

My answer:

> All cases were run in the same conditions. And they are always
> reproducible. Below are the results with USER and SYSTEM times printed
> separately:
>
> 3) f77 -64 silly.f -o silly
>
>          imax    USER            SYSTEM
>          8000  0.6193750      5.5300002E-04
>          8100  0.6530520      2.4150000E-03
 .................................................................
>
> 4) f77 -64 -O2 silly.f -o silly
>
>          8000   6.126564      4.4080000E-03
>          8100   6.200925      6.2020002E-03
 .................................................................
>
> 5) f77 -64 -O3 silly.f -o silly
>
>
>          8000  0.1880720      4.1099999E-04
>          8100  0.1910070      1.9430000E-03
 .................................................................
>
>
> As you see, results for -64 -O2 are proportionally ten-fold slower in
> both times :-(
>
>
> > I've just ran your program on a "quiet" Indigo2 R10000/195MHz
> > and this is what I got:
> >
> >        --------------------------------------------
> >          imax |     3)    |      4)   |      5)   |
> >        --------------------------------------------
> >          8000 | 0.6389320 | 0.4704070 | 0.3378320
> >          8100 | 0.6739330 | 0.4792880 | 0.3438850
> >          8200 | 0.6547930 | 0.5068420 | 0.3709340
> (..............)
>
> That certainly looks strange... Seems like the problem is specific only
> to our Indigo2's.. They all have been upgraded pretty recently to
> R10000/195MHz. And I have installed quite a number of software patches.
> Could it be a factor?
>
> Could other CCL readers reproduce these results?
>
> Robert Fraczkiewicz
> University of Texas Medical Branch
> Galveston, TX 77555
>-- End of excerpt from Robert Fraczkiewicz
===========================================================================

Similar problems were reported by other CCL readers:

On Nov 21,  9:18pm, John P. Fulmer wrote:
> Subject: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
> Hello Robert,
>         Our results were similar to yours on our SGI R10000.  Have you asked
> SGI software support about this?
> John
>
> iris6:John # 12 > silly1   silly2     silly3     silly4     silly5
>          8000  0.8272010 0.6551520  0.6201090   5.503161  0.1884650
>          8100  0.8641540 0.6898490  0.6561240   5.602162  0.1928670
>          8200  0.8729870 0.6967030  0.6354160   5.634590  0.1929920
>
>-- End of excerpt from John P. Fulmer

On Nov 22,  9:20am, Paul Verwer wrote:
> Subject: F77 compiler on R10k
> Hi Robert,
>
> We have the same problem here (sorry, I did not format the output...):
>
> f77 silly.f -o silly1
> f77 -O2 silly.f -o silly2
> f77 -64 silly.f -o silly3
> f77 -64 -O2 silly.f -o silly4
> f77 -64 -O3 silly.f -o silly5
>
> cmsc:~/src/local/silly> ./silly1
>          8000  0.8541860
 .....................................................
> cmsc:~/src/local/silly> ./silly2
>          8000  0.6700260
 .....................................................
> cmsc:~/src/local/silly> ./silly3
>          8000  0.6343770
 .....................................................
> cmsc:~/src/local/silly> ./silly4
>          8000   5.539060
 .....................................................
> cmsc:~/src/local/silly> ./silly5
>          8000  0.1926340
 .....................................................
>
> Hinv gives:
> 4 194 MHZ IP25 Processors
> CPU: MIPS R10000 Processor Chip Revision: 2.5
> FPU: MIPS R10010 Floating Point Chip Revision: 0.0
> (etc.)
> Irix is 6.2.
>
> As shown, the machine has 4 cpu's; during tests, one job was using ~100%
> cpu on one processor all the time.
> We have had big problems with this machine trying to run Cerius2; those
> appeared to be triggered by certain system patches, that were removed after
> a lot of trial and error. Maybe these patches cause the huge slow down at
-O2?
>
> Best regards,
> Paul Verwer.
> --
> Dr. P. Verwer
> CAOS/CAMM Center                        A-3014
> University of Nijmegen                  Phone: +31 (0)24 3653358
> P.O. Box 9010                           Fax:   +31 (0)24 3652977
> 6500 GL Nijmegen                        [email-masked]
> The Netherlands                         http://www.caos.kun.nl/
> ===============================================================================
>-- End of excerpt from Paul Verwer

On Nov 22,  7:20am, Marek Strajbl wrote:
> Subject: Re: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
> Hi Robert,
>
> I confirm your results.
> In the case 4) the times are 10-times longer.
> We have PowerChallenge with
>
> Processor : 194 MHZ IP25
> CPU: MIPS R10000 Processor Chip Revision: 2.4
> FPU: MIPS R10010 Floating Point Chip Revision: 0.0
>
> regards,
> 	Marek
> --
> Marek Strajbl
> Institute of Physics		phone:  (+422) 2191-1343
> Ke Karlovu 5, Praha 2 		fax:    (+422) 29 67 64
> 121 16, CZ			e-mail: <[email-masked]>
>-- End of excerpt from Marek Strajbl

On Nov 22,  9:01am, Jon Andreas Stoevneng wrote:
> Subject: Re: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
> Dear Robert,
> we have recently bought an indigo2. The output from the hinv
> command is:
>
> Iris Audio Processor: version A2 revision 1.1.0
> 1 175 MHZ IP28 Processor
> CPU: MIPS R10000 Processor Chip Revision: 2.5
> FPU: MIPS R10010 Floating Point Chip Revision: 0.0
> On-board serial ports: 2
> On-board bi-directional parallel port
> Data cache size: 32 Kbytes
> Instruction cache size: 32 Kbytes
> Secondary unified instruction/data cache size: 1 Mbyte
> Main memory size: 128 Mbytes
> EISA bus: adapter 0
> Integral Ethernet: ec0, version 1
> Integral SCSI controller 1: Version WD33C93B, revision D
> Integral SCSI controller 0: Version WD33C93B, revision D
>   Disk drive: unit 1 on SCSI controller 0
> Graphics board: Solid Impact
>
>
> I have run your program silly, and obtain results similar to yours:
>
> 1) f77 silly.f -o silly
>
>          8000  0.9095600      6.3899998E-04
 .....................................................
>
> 2) f77 -O2 silly.f -o silly
>
>          8000  0.7247770      5.0299999E-04
 .....................................................
>
> 3) f77 -64 silly.f -o silly
>
>          8000  0.6814490      8.6199999E-04
 .....................................................
>
> 4) f77 -64 -O2 silly.f -o silly
>
>          8000   6.108339      4.3990002E-03
 .....................................................
>
> 5) f77 -64 -O3 silly.f -o silly
>
>          8000  0.2068810      4.2200001E-04
 .....................................................
>
> -----------------------
>
> I guess you have a 195MHz machine? That would explain the slightly
> larger values obtained on our machine.
>
>
> Sincerely,
> Jon Andreas Stovneng
>-- End of excerpt from Jon Andreas Stoevneng

On Nov 22,  9:58am, Matthew Eldridge wrote:
> Subject: Re: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
>
> Hi Robert,
>
> Well I got very similar results to you when I tried this
> on our Challenge R10000 (194Mhz) cluster.
>
> Matt
>
> ============================================================
> Dr. Matthew D. Eldridge               Tel:+44-(0)1223 494605
> EMBL Outstation                       Fax:+44-(0)1223 494468
> European Bioinformatics Institute
> Wellcome Trust Genome Campus
> Hinxton, Cambridge CB10 1SD
> UK                          Email:[email-masked]
> ============================================================
>-- End of excerpt from Matthew Eldridge
=======================================================================

Finally, Serge Pachkovsky wrote a detailed explanation:

On Nov 22, 11:09am, Serge Pachkovsky wrote:
> This is a known bug in vanilla 6.2 f77. It is *not* actually a bug in
> Fortran compiler per se, but rather an unexpected interaction of new dynamic
> linking convenstions and compiler optimizations. Here is that happens:
>
> A bit of background: any program you compile with default options on IRIX
> (on any modern unix, for that matter) is *not* standalone. Some of the
> subroutines have to come from dynamic libraries (or Dynamic Shared Objects,
> a.k.a. DSO) at run time. In this particular case, your program has references
> to three dynamic libraries:
>
> 31> elfdump -Dl silly
>
> silly:
>
>                    **** MIPS LIBLIST INFORMATION ****
> .liblist :
> [INDEX] Timestamp               Checksum        Flags   Name
          Version
> [1]     Mar  9 13:20:05 1996    0x51f6ec32      -----   libftn.so
      sgi1.0
> [2]     Mar  9 13:16:12 1996    0xdfaae999      -----   libm.so sgi1.0
> [3]     Mar  9 18:58:37 1996    0x1f457b4c      -----   libc.so.1
      sgi1.0
>
> The actual symbol resolution happens at run time and is performed by a
> run-time linker, rld (there is a nice man page on rld on your IRIX box,
> read it - it is a good reading if you plan to use your system effectively.
> While you are at it, you might also read manual page on DSOs). All the
> references (either subroutine calls or data access) your code makes
> to symbols in other DSOs are resolved through a global offsets table (GOT,
> there is a manual page on it, as well), which basically contains an address
> of a procedure in other DSO (which you don't know at compile time) at a
> place address of which is known at the compile time. Therefore, each
cross-DSO
> subroutine call normally contains an indirect access to a GOT slot containing
> corresponding address.
>
> These adresses are filled by rld, which can operate in two modes. In
immediate
> resolution mode, rld would resolve each and every symbol at startup. However,
> most of the cross-DSO references are never used in "most" programs (your
definition
> of "most" may differ ;-). Worse, this startup-time resolution takes time.
Actually,
> if your program makes calls to half a dosen DSOs, which in turn make calls to
other
> DSOs (not unheard of in largish commercial programs), startup time resolution
> takes *quite* a time, during which the system appears to do nothing.
Therefore,
> by default, rld uses the so-called lazy resolution mode: at startup, it
replaces
> each and every subroutine pointer in GOT with a pointer to it's resolution
routine,
> which resolves the symbol the first time subroutine is called, and writes
it's
> address back into GOT, so that all the subsequent calls can go directly to
the
> intended routine.
>
> So far, so good.
>
> Since you are calling sin() and acos() (which are actually mapped to
subroutines
> __acos and __sinf in DSO /usr/lib64/libm.so) in a very tight loop, compiler
would
> like to make things happen faster, so that it loads addresses of both
functions
> in a register. However, these two functions were never called before, and GOT
> contains address of the rld's name resolution function instead. When it gets
> called, it fixed GOT entry and calls the right functions. However, the
register
> still points to the rld's name resolution routine, because rld has no way of
> knowing what this register value is supposed to mean, and can't fit it. So
> on the next iteration rld gets called again. And again. And again. And your
> program runs much slower than it should.
>
> There is an easy workaround to this. Saying:
>
> setenv LD_BIND_NOW 1
>
> at the csh prompt (or your shell's equivalent if you use something else)
would
> force rld to resolve all inter-DSO references at startup. There is also a
> patch for the problem, talk to your SGI contact person about it.
>
> Regards,
>
> /Serge.P
>-- End of excerpt from Serge Pachkovsky
===============================================================================

Omar suggested an upgrade:

On Nov 22, 10:59am, Omar G. Stradella wrote:
 .........................................................
>
> Or upgrade your Fortran compiler to 7.0 or 7.0.1. They provide
> better support for R10000 processors.
>
> Omar.
>
> --
> +---------------------------------------------------------------------+
> Omar G. Stradella, Ph.D.                    Supercomputing Applications
> Silicon Graphics, Inc.                          Computational Chemistry
> One Cabot Road, Hudson, MA 01749, USA           N 42 22'41" W 71 33'45"
> E-mail: [email-masked] Phone: +1-508-567-2258 FAX: +1-508-562-4755
> http://www.sgi.com/ChemBio                  http://reality.sgi.com/omar
> +--------  Ph-nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn  -------+
>-- End of excerpt from Omar G. Stradella
===========================================================================

The results of tests I did following Serge's suggestions:

On Nov 22,  6:42pm, Robert Fraczkiewicz wrote:
> Subject: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
> Dear Netters,
>
> Many thanks to all who replied to my query, especially to Serge Pachkovsky
> for his very informative posting. It seems that "setenv LD_BIND_NOW 1"
> is right on target, all codes work now correctly. Unfortunately, other
> binaries like jot or netscape do not work with LD_BIND_NOW set on due to
> "unresolved symbols". However, setting _RLD_ARGS to "-ignore_unresolved"
> seems to be a good workaround.
> I did RTFM pages about rld(1), elfdump(1) and dso(5) and found some useful
> options for monitoring the execution of IRIX binaries. Setting _RLD_PATH
> (or _RLD64_PATH for 64-bit executables) to /usr/lib/rld.debug
> (/usr/lib64/rld.debug) and _RLD_ARGS to "-time_summary -trace
-quickstart_info"
> allows for an insight into what really happens during program executions.
> For example, I was able to confirm existence of the f77-ld-rld bug reported
> by Serge in the case 4) of my program: rld is called back and forth at
> EVERY call to sin and acos in the loop, whereas in cases 1)-3) and 5) it is
> called only once at startup. What's interesting is that the 5)-th code is
also
> optimized, but works correctly...
>
> I have also noticed that 64-bit binaries, unlike 32-bit counterparts, fail
> to do QUICKSTART.
>
> Next, I run some of our applications optimized with -64 -O2 and found out
> that rld is unnecessarily called many times for the same routine. If some
> of you have computational chemistry programs that perform poorly on
> SGI R10000, try the test run with /usr/lib64/rld.debug ...
>
> Best regards,
> Robert Fraczkiewicz
> University of Texas Medical Branch
> Galveston, TX 77555
>
>-- End of excerpt from Robert Fraczkiewicz
===========================================================================

Peter Shenkin's opinion...

On Nov 23,  2:41pm, Peter Shenkin wrote:
> Subject: Re: CCL:Compiling F77 on SGI and Re: CCL:RPAC 11.0
>
> In my opinion, CCL is not an appropriate forum for repeated interactive
> discussions of what may or may not be a bug in a particular compiler.
>
> Certainly Robert could continue his conversation offline with Omar
> Stradella from SGI, who replied concerning some of the earlier problems
> reported.  Following resolution, it would be highly appropriate to
> post a summary to CCL.  The SGI newsgroups, on the other hand, might
> be an appropriate forum for public discussion.
>
> In any event, the problems now reported could easily be a consequence
> of the code not being 64-bit clean in some subtle way, though of
> course a compiler bug cannot be ruled out without considerably more
> detail -- which I hope CCL will not be subjected to.
>
> 	-P.
>
> --
> ****** Multicultural Holiday Song:  "I'm Dreaming of a White Kwanzaa" *****
> * Peter S. Shenkin; Chemistry, Columbia U.; 3000 Broadway, Mail Code 3153 *
> ** NY, NY  10027;  [email-masked];  (212)854-5143;  FAX: 678-9039 ***
> MacroModel WWW page: http://www.cc.columbia.edu/cu/chemistry/mmod/mmod.html
>-- End of excerpt from Peter Shenkin

 ... and my answer:

Dear Peter,

Since I highly respect your opinion, I looked again at CCL rules
(at http://www.ccl.net/ccl/welcome.html) and found out that these topics
are appropriate for CCL:

o  Reporting bugs in chemistry software and possible work- arounds
o  Questions and answers about solving computational chemistry problems
o  Hardware-related issues relevant to the computational chemistry community

Please, let me now express my opinion: compiling and linking FORTRAN programs
is a technical aspect of computational chemistry and, from my observation, it
has always been of great interest to CCL readers. It is always beneficial to
know if either purchased, or developed software compiles and executes correctly
with possibly highest performance. Sharing these experiences on the CCL forum
saves a lot time and red tape cutting spent on calling vendor supports. Even
though the subject is limited to those CCL readers who use Silicon Graphics
hardware, it is certainly more interesting than "I'm looking for an e-mail
address of Dr. X ..."

Best regards,

Robert Fraczkiewicz