Subject: CCL:Sequence Classes and CORBA Servers

From: [email-masked] (Jim Ostell)
Date: Thu, 13 Nov 1997 12:45:55 -0500

A Few Comments from NCBI

There has been quite a lot a traffic recently starting from Gerald
Loeffler's innocent question about object oriented class libraries for
interfaces to sequence data and leading to questions of NCBI's position
on CORBA and the OMG. The discussion ranged over at least three major
topics:

   How do I get programmatic access to data and tools today?
   Will a particular technology solve our problems?
   Will people actually use it (or will it really happen)?

These are often conflated together and well mixed with passion and
politics in discussions, which is why I tend to avoid them by email.
Since NCBI's opinion has been asked, I will attempt to say something on
these three topics in case it helps. Note that all three answers
represent only our own tools and opinions and are not intended as a
comprehensive discussion of the subject.

1) How do I get programmatic access to data and tools today?

Interfaces to our network servers are provided either as linkable
network client libraries in the NCBI toolkit (1), or as WWW URLs which
you can use from any WWW library (like www perl for example). These
provide all the capabilities Loeffler requested; get a sequence based
on an id, get a bunch of sequences based on a query, do a blast search,
get an alignment, etc. A small demo program, seqget.c, from the toolkit
demo directory that takes a sequence id, fetches the sequence, and
prints it in a variety of formats. There are URLs documented on our web
site that do all this as well (see "Linking to Entrez" (2) for
example). These are not only designed to be used as hot links in web
documents, but directly for querying the system from software. For the
vast majority of our system usage these provide a sufficient interface
and they are extensively used by outside sites.

2) Will a particular technology solve our problems?

None of the above was done in C++, JAVA, or CORBA, which were specified
in Loeffler's original request, although the specific capabilities he
requested are available in our tools. At NCBI we are very conservative
about our choices of technologies, since we do not want to be at their
mercy as they change. Our network servers support more than 40,000
unique users a day, running millions of queries, so we must be careful
in our approach. We prefer to focus on the task of integrating
biomedical information, doing alignments, building maps, and so on, not
on following an evolving technology. Currently for us this means C code
and relational databases as appropriate. The ASN.1 specification was
originally intended to be implemented as C++ but that language was not
very portable until the last couple years. Now that it is, we have in
fact embarked on a C++ version of the toolkit. This may serve as a more
object oriented view of our tools, and clearly it is being informed by
our experiences of the last seven years. However it has to be a part
time project given the pressure on our resources.

Over the years various computer technologies have arisen. At one time
some people suggested adopting relational databases would enable
interoperoperability. Then it was object oriented databases, PERL, TCL,
SmallTalk, and others. We, like others, are very dependent on some of
these technologies, but their adoption has not lead to deep or
widespread interoperability. This is only partly because different
technologies are appropriate in different environments. The biggest
reason is because the research domain of molecular biology is extremely
diverse and fluid. Once one moves beyond the simplest view of the data
(eg. a FASTA file) it is very difficult to find agreements that suit
everyone's needs over time, even within a single development group.
Finding ways to exchange very simple objects is not limited by the
technology choice.

The technology with by far the greatest impact that we have recently
adopted for NCBI production services is the WWW. Part of the reason it
has been so successful compared with other technologies, is that you do
not have to agree on much in the way of content for the technology to
be effective. You do not have to be much of a programmer to do useful
things with it. It has clearly demonstrated its utility and widespread
viability. As such we have put serious resources into supporting web
interfaces, and to using the web as a way of exchanging data, such as
the query and BLAST URLs I described above. Note that use of the WWW to
provide network connectivity does not restrict access to GUI
interfaces. We explicitly provide URLs that can be used for direct data
access. The advantage of using web technology itself is that we get the
advantages of all the support it is getting. Vendors are even changing
their operating systems to make them work efficiently with it, and more
capabilities are being added to web servers which we are exploiting as
they become well established. Of course all the content presented this
way is being driven by the same NCBI toolkit that supported the older
client/servers.

So I am trying to make two points here:

One is that technologies come and go in the computer industry. Only a
few have a massive and long lasting impact. Even those go through a
considerable shakeout period. We try to restrict our efforts to such
well established, central technologies, and we try to do it at least
late in their shakeout stage. We try to maximize our time spent on
biological issues, and minimize our time spent on sorting out computer
technologies.

The second point is that no matter what technology you choose, you have
to do something with it. NCBI has created a corpus of tools that
provide the infrastructure for popular and important tools as
standalone programs on many platforms and as internet client/servers.
The same routines now provide the backend power for the WWW servers.
Our infrastructure enables us to make effective use of the technology.
Should a single distributed object approach emerge from the current
technology turmoil as persuasively useful as the WWW has, we would put
the same infrastructure behind it.

3) Will people actually use it (or will it really happen)?

In my opinion, JAVA is a promising technology but is clearly in the
early shakeout stages. Our strategy here is to make our OO move to C++
while bearing in mind the principles behind JAVA. When and if JAVA
settles down we will be in a better position to take advantage of it.

CORBA is much more established and NCBI has been a member of OMG since
the very first meeting years ago. We ended up building our own
client/server because no products were available. Now that products are
available, we find them to be very heavyweight systems. In our
experience, the relatively coarse granular interactions that make
practical sense on the internet require a relatively light weight
server and www servers fill the bill nicely. For our own purposes, we
do not see any advantage to paying the technology cost of going to
CORBA.

In addition to connectivity, CORBA also offers an interface definition
language, IDL, and there is something to be said for that. IDL can also
be used as a data definition language although that is not its primary
intent. NCBI uses ASN.1, which is primarily a data definition language,
and this is how we define and encode data for our network services (be
they client/server or web based) when our own tools provide the
interface at the network. Programmers using our clients then do not
actually read and write ASN.1 directly, they use C code functions which
return C structures to them. CORBA is most commonly used the same way.
The programmer does not read and write IDL encoded information. The
CORBA compiler generates stub functions in C, C++, SmallTalk, or JAVA,
and the programmer uses those. However, IDL also defines the interfaces
as well (the access functions). When one is working with complex
objects like annotated sequences or contigs of annotated sequences or
genome assemblies of contigs, one must be careful with the default
implementation of the interfaces generated by CORBA in order to make
efficient use of bandwidth.

In order to agree on interfaces to a data model, one must first agree
on a data model. Assuming that agreement is reached, then one must
select interfaces that both take into account the practical aspects of
available technology, such as mentioned above, as well as providing
both sufficient generality and sufficient specificity to be widely
useful. As the developers of the Standard Template Libraries for C++
will attest, this is non-trivial for even objects as simple and well
understood as strings and lists.

At NCBI we elected to define the data model to provide a unifying
framework, but let the interfaces develop in response to our real time
needs to provide services to working biologists. The reasoning was that
if we could develop real tools like Entrez, BLAST, Sequin, and others,
as sets of routines developed by many different people that
interoperate closely, we would have the basis for a set of proven
useful interfaces. At this point, I would say we have very good ideas
about a relatively coarse granular interface to our services, and you
can see that in the Entrez and BLAST interfaces. We have surfaced these
interfaces in a number of ways, including ASN.1, URLs, function call
libraries, etc. However, our experience is that interfaces into the
underlying structures themselves are much more difficult to standardize
and legitimate needs of complicated tools like editors are very
difficult to generalize. We have some better ideas about this now that
we have built Sequin and the second generation Entrez as message
passing components. We are beginning discussions here about trying to
capture some of that hindsight into C++ class libraries. But we
certainly consider it premature for us to be proposing standard
interfaces at this level that we ourselves would be willing to use. We
are certainly not ready to propose such interfaces for others.

It is very obvious to us from our own usage that finding ever more
powerful and flexible ways to use our web servers is benefiting the
biomedical community at lots of levels, from end users, to web page
builders, to programmers. We continue to put heavy resources there and
to watch developments such as fastcgi programs and the use of XML as a
data description language as obviously beneficial improvements. We are
watching technologies like JAVA and CORBA but feel it is premature for
us to make a production commitment to them at this time.

Given the uncertainty regarding CORBA as the long term technology of
choice and our own skepticism about the utility of defining community
interface standards except at very simple levels right now, we do not
see the OMG standards process as the most effective path forward for
NCBI at this time. From our own perspective we need to satisfy
ourselves that we have a workable solution to our own interface needs.
While we have not made a point of going out and proselytizing our
technical approaches, we have always welcomed visitors who are
seriously interested in learning what we are doing or making use of the
tools we can offer. We are happy to share our experiences and have
offered workshops and talks at scientific meetings or at NCBI, and are
certainly willing to do so again.

                                 Jim Ostell, Ph.D.
                                 Chief, Information Engineering Branch
                                 NCBI

1) NCBI Toolkit =96 linkable C libraries for many platforms
     Anonymous ftp from ncbi.nlm.nih.gov
     In toolkit/ncbi_tools
     Current network Entrez interface is ncbi/cdromlib/accentr.h
     Current BLAST interface (network or standalone) ncbi/tools/blast.h
     Examples of usage in ncbi/demo
     Hardcopy documentation (real old..sorry) by request from
[email-masked]

2) URL access =96
     NCBI home page =96 http://www.ncbi.nlm.nih.gov/
     Linking to Entrez =96 =
http://www.ncbi.nlm.nih.gov/Entrez/Linking.html
     Linking to BLAST =96 on our anonymous ftp site, look in
blast/blasturl/README