Subject: XML as program output?

From: "Scott Dixon" <[email-masked]>
Date: Tue, 18 May 1999 19:40:00 -0400

>
> This is a pointless discussion.  I'm seeing two main threads:
>

I don't think this is a pointless discussion at all.  As one who has read the
XML specs and written XML readers and writers, let me make a few points and
correct a few misunderstandings about XML.  First of all, XML is not "just
another format".  It is a language and standard to allow specification and
design of data interchange languages.  It is not about markup (i. e. XML is not
another HTML) although another related proposed standard (XSL) allows for
specification of markup, independent of the content markup (in XML).  That is,
>from a web point of view, one of the great advantages of XML over HTML is that
XML separates semantic markup from display markup.  However, for the purposes of
this discussion that isn't all that important, except for the fact that there
will be tools which will display XML in easy to read ways (e. g. browsers).

> A) Adding vs removing tags:
> The presence or absence of tags is irrelevent; the only issue is how
> well-formatted the text is to start with.
> o  It is easy to add tags to well-formatted raw text
> o  It is easy to remove tags from well-formatted tagged text
> o  It is difficult to add tags to poorly-formatted raw text
> o  It is difficult to remove tags from poorly-formatted tagged text
> (Spoken from the perspective of someone who spent the last several years
> doing all four).  XML would be appealing if it required that well-formatted
> files.  Unfortunately, it doesn't, or at least it doesn't any more than
> HTML does.  Look at all the crappy HTML that is out there.  Look at the
> garbage produced by PageMill or FrontPage or most any other commercial
> program.  Look through release notes for earlier versions of Netscape or
> MSIE and see the hoops those developers had to go through to deal with all
> the creative html that people cooked up.  All arguments I've seen for XML
> take it as an implicit fact that XML-compliant files will magically be
> well-formatted and easily understandable.  Guess what folks; it ain't gonna
> happen.

I'm not sure what "well-formatted" means in this context, but using HTML as an
example is not very useful.  XML, as opposed to HTL, has a way to specify
exactly what is legal and what is not (the DTD or Document Type Declaration).
It is possible to write legal XML which does not have a DTD, but even so, the
sort of uncontrolled changes which occurred in HTML are not necessary in XML
since it, as the name implies, is inherently extensible.  This, in my mind, is
the important advantage of XML and other self-defining formats.  They allow for
extension of a file format without breaking anything.  For example, one of the
problems with many text formats is that there is no place to put certain types
of information.  For example, there is no way to specify bond orders in PDB
files.  So if you want to write a molecule out in PDB format, you simply lose
that information.  However, it is possible to add another tag to an XML file and
any program which parses XML can still parse that extended file.  The
information is not lost, although some programs which read the file may not know
what to do with it.  If you add bond type to a PDB file, then it is no longer a
PDB file and some programs which read PDB files may fail to read your extended
file format.  Also, see below about name spaces.
>
> B) Text vs non-text
> This is even more pointless.  "Text" is a description of the character set
> used, and not even a very good description.  There is no such thing as a
> program that "understands text" any more than there is a person who
> "understands speech".  Traditionally, "text" files are written with the 96
> ASCII characters from 32 to 127.  The Gaussian file format is one such file
> type.  It is possible to say that "a program interprets the Gaussian file
> format."  Saying so does not in any way imply that it will interpret the
> MOPAC file format or the PDB file format  -- any more than saying that
> someone understands English implies that he also understands French.  XML
> files *are* text in that they are written using those 96 printable ASCII
> characters.  So are HTML files.  So is this message.  So what?

Actually. XML is not based on ASCII text but on Unicode, which means that it is
not English language centric (a minor point to those of us in the US but it
illustrates that the XML developers thought a lot more about how to do data
interchange correctly that does the average designer of a file format).
>
> Programmatically, "text" files are often *less* easy to interpret than
> binary files.  There are lots of benefits to using binary file format,
> including density of data and speed of interpretation.  The major drawback
> is that they are not readily interpretable by unassisted humans.

There is also the problem that binary formats are machine specific with respect
to byte order, number representation, etc.  They do not make good interchange
formats.  They may be fine as an internal format used within a program but they
are a poor choice for data interchange or archival.
>
> XML is "just another file format".  There is no program in the world that
> will magically be able to interpret XML without specific programming
> effort.  The programming effort to interpret XML is neither significantly
> more nor significantly less than that required to interpret any other file
> format (assuming a similar number of data *types* in both cases).

This is the crux of the matter I think.  To interpret (not parse or read) an XML
file requires a common understanding about the meaning of the tags used.  That
is, if my XML file specifies X, Y and Z coordinates, the reader of the file
still has to know that they are atomic coordinates in angstroms, for example.
However, this is a issue which is dealt with in XML in a couple of ways.  First
of all, there is the namespace facility in XML which allows definitions of tags
to be published and used.  For example, the tag <scott:x> refers to an x
datatype, the definition of which I have published at a specified URI on the
net.  If someone else also uses the tag <xtal:x> it may mean an x coordinate in
a unit cell.  Both of these tags can be used in an XML file and the definitions
can be referred to at published URIs.  So there is a way to deal with
conflicting definitions by specifying which one you mean or using both as
appropriate.  Also, the RDF (Resource Description Framework) is worth looking at
(www.x3c.org/RDF) with respect to this issue.  Nevertheless, it is true that XML
will not magically solve all the problems with conflicting uses of terms and
poorly specified concepts in chemical information.  XML does not obviate the
necessity for people to define what they mean.  There is no such thing as a free
lunch.  Those software designers who don't want to play nice can still make life
difficult for the rest of us.  But, the designers of XML have given a lot of
thought to issues of data interchange and metadata and it is worth looking hard
at XML for its advantages of easy parsing, extensibility and self definition.
The issue of agreeing on a commonly understood vocabulary still remains but at
least with XML, there are well defined ways to deal with vocabulary definition
and multiple vocabularies.  Most importantly, XML allows one to extend a file
format or add to a vocabulary without breaking the ability of everyone else to
read that file.
I notice that while I have been typing this, Henry Rzepa has also responded to
many of the same issues so I'll stop here.  There are a lot of other interesting
issues concerning data interchange and metadata which deserve more attention
than they have generally gotten from the designers of chemical file formats.
There is a great deal of information about XML available on the Web.  Henry
mentioned some resources and the canonical XML site is www.w3c.org.

Scott Dixon
Metaphorics, LLC

---

From: Gerardo González Aguilar <[email-masked]>
Date: Wed, 19 May 1999 08:34:54 -0400 (CDT)

Hi Jonathan

I am agree with your opinion, really it is usefull and standard code for
chemistry, and as you wrote converters have the problem of loss of
information, if these code is xml or any other is a minor problem, the
principal task will be to implemment teh output of the currently
used programs in the standard format, the programmers of these packages  
may be are disagreed with the idea and in that case and uniform output for
all the chemistry programs will be a fantasy.

Bye...

Gerardo

-------------------------OOOooooOOO-------------------------------------
                   |                        | e-mail:
Gerardo Gonzalez   |  Dpto. de Quimica      | [email-masked]
Prof. of Chemistry |  Univ. de Camaguey     | or:
                   |  C. Circunv km 5 1/2   | [email-masked]
                   |  Camaguey 74650, Cuba  |
------------------------<>--------<>-------------------------------------

---

From: Peter Shenkin <[email-masked]>
Date: Wed, 19 May 1999 12:23:39 -0400 (EDT)

Hi,

I've been enjoying this discussion.  Here's an essay on the
"big picture", as I see it.

There have been several extensible file formats and/or APIs
proposed and used in chemical applications over the past few 
years.  These include:

	asn.1: Finds its greatest use in telecommunications, but is
	used by NIH/NCBI to encode both genetic and chemical-structural
	data.

	STAR and STAR/CIF: STAR was originally proposed in a chemical
	context;  STAR/CIF both limits the syntax of STAR and defines
	a data dictionary with semantics useful to crystallographers, 
	expecially small-molecule crystallographers.

	XML and XML/CML: The latter, as I understand it, attempts to 
	define types and semantics for types that will be useful for
	chemists.  But I know least about this one, especially today.

	CEX: This never really took off, but is an API which Dave
	Weininger of Daylight Software proposed for exchange of chemical
	data.

We are going to be distributing a new file format, called the m2io 
format, with our new MacroModel interface, due out this summer.
All the schrodinger products will be able to read this format and
we will probably make the spec and the API to read/write it publically
available, as we did with our old mmio format (whose API code happily
resides in the CCL archive :-) ).  However, we want to get some field
experience with m2io first.  The m2io format will share some of the 
features of those listed above.

All of the above have well-defined, extensible syntaxes.  So if tomorrow
you decide you need to store, say, two partial charges for each atom
instead of one, you can do it without worrying about forward/back
compatibility. (Why two charges?  Well, maybe one comes from ab-initio
computations and the other comes from a forcefield assignment.  Why
in the same file?  Well, maybe you have a graphics interface and you
want to color-code the atoms where these two differ the most.)

The point about the extensible nature of these formats is that, with
a few exceptions, if you have data in one of these formats, it
is easy in principle to convert it to another one -- including even
the extended features. 

One difference among these formats is the degree of nesting of
data structures they allow.  ASN.1 allows as many as one wants;  
so does STAR.  But STAR/CIF limits this, and CEX started out with 
basically the STAR/CIF limits.  I'm not sure about the XML/CML
proposal, but I suppose it allows as much as you want.  The m2io
format allows infinite nesting, but the API only supports a limited
degree right now, like CEX and STAR/CIF.

The real difficulty, however, is how semantics are to be handled.  CEX
had a field called CHARGE.  In CEX work-group discussusions, I used
to ask, "Is this partial charge or formal charge."  The answer was
always, "It's whatever you decide to put there."  The net effect
was that a reading program would not always know how to interpret
CHARGE when it got this field.  IMHO, it was a failure to resolve
this sort of issue that limited the success of CEX.

It would be easy to say, "Well, if I put the partial charge in
the CHARGE field and read it in one of my programs later, then
I know it's partial charge and it's not a problem."  But if these
APIs and file formats are to be useful for data interchange with
someone else's program, I need to know what the other program
meant by CHARGE.  That means I need not just the file, but 
also the history of its generation.  There was no standard
way of encoding this.

How do the other APIs/formats handle the semantics of the data
tags?  Well, one approach is to set up a committee to define,
as precisely as possible, what the different tags (like CHARGE)
are going to mean.  This is the CIF and, AFAIK, the CML approach.
My understanding is that CIF, at least, has been terribly successful
within its application domain.

The concern I have about this approach is, "Would the 
committee's definitions meet my own needs?".  Also, life is
easier when one is working in a narrow domain, as CIF
does.  There, the committee could use international 
crystallographic conventions for tagging space groups and 
so on, and I'm sure the crystallographers had no trouble
agreeing on what a fractional coordinate is.  

But when we attempt to deal with the broader field
of chemistry in general, where we have ambiguities as well
as subtleties, defining semantics for a long list of tags 
is going to be much harder.  What are the semantics
of "a hydrogen bond" or "an alpha helix"?  A lot of the 
difficulties in using PDB files have to do with this.  Of
course, a lot also have to do with the format's inability to 
encode far simpler things whose semantics we would have less 
trouble agreeing on.

ASN.1 is widely used in a variety of separate, narrow contexts
in the telecommunications field.  The data definitions ("abstract
syntaxes") were for the most part derived by committees or in
in response to RFPs within these areas.  The NIH/NCBI abstract
syntax definitions, in contrast, were devised within NIH and
promulgated to users of their databases and software.  We
took a very close look at using their biostructural data 
definitions, but eventually determined that they were not
well-suited to the encoding of general chemical structures;
there was a built-in residue-based bias, appropriate to 
biopolymers but difficult to extend to other things.

We then devised our own ASN.1 abstract syntax and even build
molecules using it.  But we abandoned this approach because we wanted
a file format that would be more readable than any ASN.1 format
(even the ascii "value notation") provides.   IO was ponderous
and the format was verbose.

Let tell you how we're dealing with the semantic issue in m2io.
Part of each data tag may contain a label that identifies a 
"reader", sort of an intended audience.  

For instance, we have a broader API that uses m2io called mmct,
which manages ct's, or connection tables (assemblies of molecules).
Some of our data tags contain an "mmct" designation;  for example, in
effect, "mmct_x" says "this is 'x' in the sense that mmct
understands it".  (For instance, it might be an atomic x
coordinate.)  

Depending how well you feel you understand what mmct does,
you may use each such item freely in your own program.  You may,
of course, also add to the file new data for your own reader;
say, "foo_x", which might mean "x as foo understands it."  Of 
course, anybody is free to read or write anybody else's data types.  

Why did we roll our own, instead of using one of the above choices?
Well, we did come close to using ASN.1, as I mentioned above.  For
STAR and XML/CML, we couldn't find readers/parsers that were either
free or cheap enough -- or, if I recall correctly, for CML, that even
existed.  The good ones for ASN.1 are expensive, too.  All the above
standards and emerging standards are large.  Just understanding
them well enough to write our own parser correctly would have 
been a huge task -- perhaps larger, even, than that of actually 
writing the parser.

We could have used our reader-encoding system in CEX tags of our
own devising, but we wanted something with a defined file format;
CEX is an API only.  If CEX had caught on among others, perhaps 
we would have gone with it and perhaps added a file format rather
like m2io to CEX.  But in the end we found it easier to start 
from scratch.  The great thing about wheels is that they can be 
so easily reinvented. :-)

But if one of the other formats -- say, CML -- becomes popular, and
if API code becomes widely available, it will be easy to write 
translators to and from m2io.  In the meantime, we've come up
with something that we feel will be very useful to us, and hopefully,
in time, to others as well.

	-P.

--
********* Peter S. Shenkin; Schrodinger, Inc.; (201)433-2014 x111 *********
*********** [email-masked]; http://www.schrodinger.com ***********