Subject: Re: CCL:F77 compiler on R10000 SGI Indigo2: buggy?

From: [email-masked] (Serge Pachkovsky)
Date: Fri, 22 Nov 1996 11:09:20 +0100 (MET)

Hi,

> 1) f77 silly.f -o silly        (this produces 32-bit mips2 code by default)
> 2) f77 -O2 silly.f -o silly
> 3) f77 -64 silly.f -o silly    (this produces 64-bit mips4 code)
> 4) f77 -64 -O2 silly.f -o silly
> 5) f77 -64 -O3 silly.f -o silly

> The unusual results I have obtained are shown in table below. Apart
> from the fact that the 64-bit executables produce slightly different
> result for x than their 32-bit counterparts, the CPU times in column
> 4) are larger by a factor of TEN in comparison to other columns!
> Normally, I would expect them to be in between 3) and 5). This abnormal

This is a known bug in vanilla 6.2 f77. It is *not* actually a bug in
Fortran compiler per se, but rather an unexpected interaction of new dynamic
linking convenstions and compiler optimizations. Here is that happens:

A bit of background: any program you compile with default options on IRIX
(on any modern unix, for that matter) is *not* standalone. Some of the
subroutines have to come from dynamic libraries (or Dynamic Shared Objects,
a.k.a. DSO) at run time. In this particular case, your program has references 
to three dynamic libraries:

31> elfdump -Dl silly

silly:

                   **** MIPS LIBLIST INFORMATION ****
 .liblist :
[INDEX] Timestamp               Checksum        Flags   Name            Version
[1]     Mar  9 13:20:05 1996    0x51f6ec32      -----   libftn.so       sgi1.0
[2]     Mar  9 13:16:12 1996    0xdfaae999      -----   libm.so sgi1.0
[3]     Mar  9 18:58:37 1996    0x1f457b4c      -----   libc.so.1       sgi1.0

The actual symbol resolution happens at run time and is performed by a
run-time linker, rld (there is a nice man page on rld on your IRIX box,
read it - it is a good reading if you plan to use your system effectively.
While you are at it, you might also read manual page on DSOs). All the 
references (either subroutine calls or data access) your code makes
to symbols in other DSOs are resolved through a global offsets table (GOT,
there is a manual page on it, as well), which basically contains an address
of a procedure in other DSO (which you don't know at compile time) at a
place address of which is known at the compile time. Therefore, each cross-DSO
subroutine call normally contains an indirect access to a GOT slot containing
corresponding address. 

These adresses are filled by rld, which can operate in two modes. In immediate
resolution mode, rld would resolve each and every symbol at startup. However,
most of the cross-DSO references are never used in "most" programs (your definition
of "most" may differ ;-). Worse, this startup-time resolution takes time. Actually,
if your program makes calls to half a dosen DSOs, which in turn make calls to other
DSOs (not unheard of in largish commercial programs), startup time resolution
takes *quite* a time, during which the system appears to do nothing. Therefore,
by default, rld uses the so-called lazy resolution mode: at startup, it replaces
each and every subroutine pointer in GOT with a pointer to it's resolution routine,
which resolves the symbol the first time subroutine is called, and writes it's
address back into GOT, so that all the subsequent calls can go directly to the
intended routine.

So far, so good.

Since you are calling sin() and acos() (which are actually mapped to subroutines
__acos and __sinf in DSO /usr/lib64/libm.so) in a very tight loop, compiler would
like to make things happen faster, so that it loads addresses of both functions
in a register. However, these two functions were never called before, and GOT
contains address of the rld's name resolution function instead. When it gets
called, it fixed GOT entry and calls the right functions. However, the register
still points to the rld's name resolution routine, because rld has no way of
knowing what this register value is supposed to mean, and can't fit it. So
on the next iteration rld gets called again. And again. And again. And your
program runs much slower than it should.

There is an easy workaround to this. Saying:

setenv LD_BIND_NOW 1

at the csh prompt (or your shell's equivalent if you use something else) would
force rld to resolve all inter-DSO references at startup. There is also a
patch for the problem, talk to your SGI contact person about it.

Regards,

/Serge.P

---

From: "Omar G. Stradella" <[email-masked]>
Date: Fri, 22 Nov 1996 10:59:12 -0500

On Nov 22, 11:09am, Serge Pachkovsky wrote:
> Subject: CCL:F77 compiler on R10000 SGI Indigo2: buggy?
> Hi,
>

[...]

> patch for the problem, talk to your SGI contact person about it.

Or upgrade your Fortran compiler to 7.0 or 7.0.1. They provide
better support for R10000 processors.

Omar.

-- 
+---------------------------------------------------------------------+
Omar G. Stradella, Ph.D.                    Supercomputing Applications 
Silicon Graphics, Inc.                          Computational Chemistry
One Cabot Road, Hudson, MA 01749, USA           N 42 22'41" W 71 33'45"
E-mail: [email-masked] Phone: +1-508-567-2258 FAX: +1-508-562-4755 
http://www.sgi.com/ChemBio                  http://reality.sgi.com/omar 
+--------  Ph-nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn  -------+

---

From: "Robert Fraczkiewicz" <[email-masked]>
Date: Fri, 22 Nov 1996 18:42:08 -0600

Dear Netters,

Many thanks to all who replied to my query, especially to Serge Pachkovsky
for his very informative posting. It seems that "setenv LD_BIND_NOW 1"
is right on target, all codes work now correctly. Unfortunately, other
binaries like jot or netscape do not work with LD_BIND_NOW set on due to
"unresolved symbols". However, setting _RLD_ARGS to "-ignore_unresolved"
seems to be a good workaround.
I did RTFM pages about rld(1), elfdump(1) and dso(5) and found some useful
options for monitoring the execution of IRIX binaries. Setting _RLD_PATH
(or _RLD64_PATH for 64-bit executables) to /usr/lib/rld.debug
(/usr/lib64/rld.debug) and _RLD_ARGS to "-time_summary -trace -quickstart_info"
allows for an insight into what really happens during program executions.
For example, I was able to confirm existence of the f77-ld-rld bug reported
by Serge in the case 4) of my program: rld is called back and forth at
EVERY call to sin and acos in the loop, whereas in cases 1)-3) and 5) it is
called only once at startup. What's interesting is that the 5)-th code is also
optimized, but works correctly...

I have also noticed that 64-bit binaries, unlike 32-bit counterparts, fail
to do QUICKSTART.

Next, I run some of our applications optimized with -64 -O2 and found out
that rld is unnecessarily called many times for the same routine. If some
of you have computational chemistry programs that perform poorly on
SGI R10000, try the test run with /usr/lib64/rld.debug ...

Best regards,
Robert Fraczkiewicz
University of Texas Medical Branch
Galveston, TX 77555