Subject: XML as program output?

From: "David E. Bernholdt" <[email-masked]>
Date: Tue, 18 May 1999 12:56:10 -0400

I'm curious what people think about using XML for the output of
chemistry programs?  I'd appreciate your response to the scenarios I
present below.

Lets imagine its a year or two into the future, and you have an
XML-capable browser that's as loaded with features as your current
HTML browser.

You run a chemistry code -- say an SCF calculation, to be concrete.
Would you prefer that the primary output of this program be in plain
text, as nearly all packages I'm familiar with now produce, or would
you prefer it to be thoroughly marked-up XML?

The plain text version may be more compact if you page through it
using more, or look at it in a text editor, but if you take the XML
version, you have all the capabilities of the browser to help you.
You could apply different style sheets to highlight different parts of
the output.  By judicious use of styles & applets, you might even be
able to have the molecular geometry rendered with a 3-d viewer, or
visualize orbitals, etc.

Do the increased presentational possibilities made possible by using
XML offset the additional complexity/length of the output file and the
need to have an XML browser to view it properly?

Now consider the fact that you could also use commodity XML tools to
parse the output file & extract specific items for further processing
-- perhaps to be stored into a database, a spreadsheet, etc.

Does this make XML worthwhile as the primary output format of a
chemistry package?

How about if the XML conformed to a standardized schema that was used
by _all_ SCF programs, so that they all marked up their output the
same way?  How about if all electronic structure programs conformed to
the same schema?  Or the unlikely possibility that there was a schema
which encompassed "all" chemistry software?

What would it take, in your opinion, for XML-based output to be
preferrable to the current-day plain text output?

Thanks for your thoughts.
--
David E. Bernholdt                      | Email:  [email-masked]
Northeast Parallel Architectures Center | Phone:  +1 315 443 3857
111 College Place, Syracuse University  | Fax:    +1 315 443 1973
Syracuse, NY 13244-4100                 | URL:    http://www.npac.syr.edu

---

From: Richard Wood <[email-masked]>
Date: Tue, 18 May 1999 11:56:11 +36000

I think text is preferable to any other format, IMO, now or in the future.
If one needs xml or another format, one can just add the appropriate tags 
as needed.

On Tue, 18 May 1999, David E. Bernholdt wrote:

> -= This is automatically added to each message by mailing script =-
> [email-masked] -- To Everybody    |   [email-masked] -- To Admins
> [email-masked] -- HELP CHEMISTRY or HELP SEARCH
> [email-masked] -- archive search    |    Gopher: gopher.ccl.net 70
> Ftp: ftp.ccl.net  |  WWW: http://www.ccl.net/chemistry/   | Jan: [email-masked]
> 
> 
> 
> 
>

---

From: Gerald Loeffler <[email-masked]>
Date: Tue, 18 May 1999 21:19:25 +0200

Hi chemistry-XMLers !

David presents an excellent example, although, IMHO his phrasing is
misleading. The _real_ question is whether to go for XML-conformant
text-output or for unstructured/arbitrarily structured text-output of
chemistry programs. Because XML code _is_ text, i.e. by simply using a
plain text editor XML code is as readable and understandable by humans
as any other text-output can be. It is only for visually pleasing
presentation that you need a helper-application such as a XML-capable
web-browser!

The _real_ advantage of XML over arbitrarily structured text-output is
the fact that XML is easily parseable and, what's more, the
meta-information on how to legally form a specific class of
XML-documents (e.g. the SCF-output of a Gaussian-run) is standardized
and easily parseable as well.

These advantages don't even require a commonly agreed-upon format for
equivalent information. I.e. Gaussian and its competitors might (and
probably will (-: ) output SCF-results as XML-documents of a different
type. The interconversion, however, is much more straightforward than
now, because 1) the parsing-problem does not exist and 2) there is a
strict definition of how the different formats may look like.

	just my $0.02,
	gerald

-- 
 Gerald Loeffler
 Email: [email-masked]
 Smail: Apollo Imaging, Marchettigasse 7, A-1060 Vienna, Austria
 Phone: +43 676 3289588 (+43 1 5952333 27)
 Fax:   +43 1 5952333 20
 Keywords: Java, CORBA, OOA&D, Databases, Bioinformatics,
           Computational Biology, Computational Biophysics

---

From: David van der Spoel <[email-masked]>
Date: Tue, 18 May 1999 21:25:58 +0200 (MEST)

>If one needs xml or another format, one can just add the appropriate tags 
Text files are usually good for experts, they can interpret output files
by reading them. Moreover you can easily post-process text files using
small programs or scripts in any language you like.
However there is very often reason to visualize results, since in many
cases it will give more understanding, but also because some things are
hard to grasp from a text file. I prefer to look at a molecule, however
small, using rasmol rather than look at the coordinates. When checking the
geometry generated using whatever, it is much easier to look at a
printed list of bond length and angles than clicking on the atoms in a
GUI. Thus I think we want the text files as well as something else, and
that something else could well be XML (I don't know it). To have any
chance at all to be accepted as "standard" I think the following demands
have to be met:
- Chemistry/XML format should be non proprietary, i.e. it may not be
  crippled by browser vendors (like HTML). Ideally the format should be
  controlled by chemists
- There should be a free (GPL'ed) library which programmers can use to
  write xml output (callable from at least C and fortran, meaning that it
  should *not* be a C++ library).
- There need to be fast portable browsers

Looking on the web I found the SGML/XML home page at
http://www.oasis-open.org/cover/sgml-xml.html

Groeten, David.
________________________________________________________________________
Dr. David van der Spoel		Biomedical center, Dept. of Biochemistry
s-mail:	Husargatan 3, Box 576,  75123 Uppsala, Sweden
e-mail: [email-masked]	www: http://zorn.bmc.uu.se/~spoel
phone:	46 18 471 4205		fax: 46 18 511 755
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

---

From: Richard Wood <[email-masked]>
Date: Tue, 18 May 1999 13:48:18 +36000

But rasmol reads a "text" file format!  Why change it and other programs 
that already read text files?  That doesn't make any sense to me.

Richard

---

From: "Wolf-Dietrich Ihlenfeldt" <[email-masked]>
Date: Tue, 18 May 1999 21:35:21 -0600

On May 18, 11:56, Richard Wood wrote:
> Subject: CCL:XML as program output?

I strongly disagree with this view!!!!

It is always easy to
strip off tags by a simple procedure, but the automatic addition of tags
is often not possible, and manual addition tedious and impractical.

>From the standpoint of chemical information processing, there are
probably no worse output formats than those written by many QM packages.
Anybody who had to adjust scripts which extract some value from
these text dumps time after time after a new program version was released,
or after some yet unencountered condition led
to ever so subtle reformatting of the output, will agree.
Output should always be designed to
be readable by other programs, and this requires a clearly
defined structure. It should be possible to
identify the meaning of every piece of output, without
human guessing. This is what the tags
are primarily for: Not for formatting, but
for content description. Many of the standard exchange formats, designed
decades ago, such as MDL Molfiles or PDB files, are atrocious with
respect to clean design, avoidance of unnecessary limitations
and general expressiveness, but they are still far superior to
what programs such as Gaussian throw at us.

XML is definitely the way to go, because of its general-purpose, flexible
design and widespread support (next browser generation,
next MS Office generation, plus lots of free tools are available already,
for example from IBM alphaworks).

By the way, this is not the first time XML has
been proposed  as chemistry exchange format.
Pioneering work was carried out by by Peter
Murray-Rust and H. Rzepa on this field. Have a look
at www.xml-cml.org for the present status. A first
definition of a chemical DTD (document type definition)
(CML - Chemical Markup Language) is available,
and also an experimental Java Browser for these files.

--
Dr. Wolf-D. Ihlenfeldt
Computer Chemistry Center, University of Erlangen-Nuernberg
Naegelsbachstrasse 25, D-91052 Erlangen (Germany)
Tel (+49)-(0)9131-85-26579  Fax (+49)-(0)9131-85-26566
---
The three proven methods for ultimate success and fame:
1) Nakanu nara koroshite shimae hototogisu
2) Nakanu nara nakasete miseyou hototogisu
3) Nakanu nara naku made matou hototogisu

---

From: Richard Wood <[email-masked]>
Date: Tue, 18 May 1999 13:46:37 +36000

> is often not possible, and manual addition tedious and impractical.

i think it's just as easy to ADD tags as it is to remove them.  text 
files have the advantage in that they can EASILY be imported into most 
other applications.

Also, to change the output formats of dozens of programs to a format that 
is not very widely used, especially considering that not everyone is 
using the same OS and same versions of programs, is daft.

Richard

---

From: Dmitry Khoroshun <[email-masked]>
Date: Tue, 18 May 1999 16:01:14 -0400 (EDT)

> that already read text files?  That doesn't make any sense to me.

This indeed makes not much sense. What use is of introducing a "type"
of the energy, like "nuclear repulsion"? You still have to read and
understand it, not just click on it with a mouse. Unix scripting
possibilities allow for many many things like sorting or similar. And of course
programs such as molden have no problems with reading text files.

I guess the next step would be implementing surrounding sound into Gaussian.

Sincerely,
Dmitry Khoroshun
[email-masked]

---

From: "Wolf-Dietrich Ihlenfeldt" <[email-masked]>
Date: Tue, 18 May 1999 22:16:53 -0600

On May 18, 13:46, Richard Wood wrote:
> Subject: Re: CCL:XML as program output?
>
> i think it's just as easy to ADD tags as it is to remove them.

No, this is certainly not true. I doubt that anybody really
wants to do this. We are not talking
about something simple like adding an HTML header and a few
paragraph breaks for a quick publication of a text as
a web page, but about a careful markup, identifying and
classifying each and every number, procedure, etc.

 text
> files have the advantage in that they can EASILY be imported into most

Not true either. Into what applications? Word processors? Maybe.
But who wants to do that?  If you count the number
of applications which can read a PDB file, compared
to those who are able to understand the typical textual
output of some popular QM application (with all its variants,
platform-dependend formatting varations, version changes, etc.)
you will have to agree that is is a very good thing to have
a standard.
> using the same OS and same versions of programs, is daft.

Better change them now, instead of adding another undocumented
output variation because you hacked some additional feature,
or change a column width because your new 64-bit machine gives you better
precision.
The idea of tagging is that the second set of programs,
which process the output of the first set,  can easily
extract from the data the part they can handle, without having
to understand all intricacies of a text dump, using only
one XML  parser. If new output data is encapsulated into
its own tags, it is easy to ignore. Adding another
column to a text dump is very dangerous, because it will
desynchronize scripts which cannot do anything
more intelligent than counting columns
until they  find their data. This has nothing to do
with platform independence. XML IS platform-independent,
since it is ASCII and does not suffer from  any byte ordering
or FP representation issues.

---

From: Richard Wood <[email-masked]>
Date: Tue, 18 May 1999 14:27:57 +36000

But isn't typing <energy="     "     > as difficult as typing <img 
src="mypicture.gif">?

On Tue, 18 May 1999, Wolf-Dietrich Ihlenfeldt wrote:

> But who wants to do that?  

A lot of people do, myself included.  Does MS Excel read xml files?  No, 
yet it reads text files.

> If you count the number

I don't use pdb files.  PDB files are generally useless because of their 
lack of connectivity tables, at least in the programs i use.

---

From: Richard Wood <[email-masked]>
Date: Tue, 18 May 1999 14:56:09 +36000

Well, the notion of coming up with a NEW file format to replace 10++ file 
formats confuses me even more.

Richard

---

From: "Jesus M. Castagnetto" <[email-masked]>
Date: Tue, 18 May 1999 14:00:34 -0700 (PDT)

Richard and all,

It is not more that you need to change the internal format of programs like
Rasmol. Think of XML as a data encapsulation format, which can be filtered
to any other proprietary or widely used format. The idea of XML is to be
not only markup, but that the markup will have meaning.

So XML is not only for information about the structure of a document, it can
also be treated as data.
Two of the examples of XML application I know are:

WDDX: http://www.wddx.org/ (Wed Distributed Data eXchange)
CML: http://www.xml-cml.org/ (Chemical Markup Language)

The latter has been for about 2 years already and there is ever a CML/XML
java browser (and sort of editor) by one of the proponents of CML (Peter
Murray-Rust) named "Jumbo".

Now XML is not only a text file format, if contains meta-information about
the data in it. And if your have a DTD (Document Type Definition) for the
markup, the validity of the information/structure can be validated too.

There is a great push for using XML and Java, or uxing XML as a CORBA
container. Imagine you have an agent program that understands XML, you can
ask it to go and fetch information from dissimilar sources, and then put it
into XML, which then can be both displayed on your computer and put into
an XML-object into a database (or translated into an SQL string for an
SQL database).

Or maybe think of XML as a general description format from which you can
translate to/from to your favorite program.

On Tue, 18 May 1999, Richard Wood wrote:

> Date: Tue, 18 May 1999 13:48:18 +36000
> From: Richard Wood <[email-masked]>
> To: David van der Spoel <[email-masked]>
> Cc: "David E. Bernholdt" <[email-masked]>, [email-masked]

Hope I dit not bore you to death with my rant.

Best regards.

--
Jesus M. Castagnetto, PhD <[email-masked]> - "Ken Zen Ichi-nyo"
Program Project:   http://www.scripps.edu/research/metallo/ 
Metalloprotein DB: http://metallo.scripps.edu/
Pilot Stuff: http://www.geocities.com/ResearchTriangle/Lab/1059/

---

From: Richard Wood <[email-masked]>
Date: Tue, 18 May 1999 15:05:35 +36000

It just seems like someone created a "new" file format without any apps 
to use it and is now trying to ram it down our throats.

Richard

---

From: Jonathan Brecher <[email-masked]>
Date: Tue, 18 May 1999 16:55:38 -0400

>files have the advantage in that they can EASILY be imported into most

This is a pointless discussion.  I'm seeing two main threads:

A) Adding vs removing tags:
The presence or absence of tags is irrelevent; the only issue is how
well-formatted the text is to start with.
o  It is easy to add tags to well-formatted raw text
o  It is easy to remove tags from well-formatted tagged text
o  It is difficult to add tags to poorly-formatted raw text
o  It is difficult to remove tags from poorly-formatted tagged text
(Spoken from the perspective of someone who spent the last several years
doing all four).  XML would be appealing if it required that well-formatted
files.  Unfortunately, it doesn't, or at least it doesn't any more than
HTML does.  Look at all the crappy HTML that is out there.  Look at the
garbage produced by PageMill or FrontPage or most any other commercial
program.  Look through release notes for earlier versions of Netscape or
MSIE and see the hoops those developers had to go through to deal with all
the creative html that people cooked up.  All arguments I've seen for XML
take it as an implicit fact that XML-compliant files will magically be
well-formatted and easily understandable.  Guess what folks; it ain't gonna
happen.

B) Text vs non-text
This is even more pointless.  "Text" is a description of the character set
used, and not even a very good description.  There is no such thing as a
program that "understands text" any more than there is a person who
"understands speech".  Traditionally, "text" files are written with the 96
ASCII characters from 32 to 127.  The Gaussian file format is one such file
type.  It is possible to say that "a program interprets the Gaussian file
format."  Saying so does not in any way imply that it will interpret the
MOPAC file format or the PDB file format  -- any more than saying that
someone understands English implies that he also understands French.  XML
files *are* text in that they are written using those 96 printable ASCII
characters.  So are HTML files.  So is this message.  So what?

Programmatically, "text" files are often *less* easy to interpret than
binary files.  There are lots of benefits to using binary file format,
including density of data and speed of interpretation.  The major drawback
is that they are not readily interpretable by unassisted humans.

XML is "just another file format".  There is no program in the world that
will magically be able to interpret XML without specific programming
effort.  The programming effort to interpret XML is neither significantly
more nor significantly less than that required to interpret any other file
format (assuming a similar number of data *types* in both cases).

The major benefit of XML -- or any other file format -- would be in how
widely it is accepted.  Converting between file formats is a bitch, and
often involves some sort of data loss.  *IF* (and it's a big if) XML
becomes widespread, it would be a big win for everyone.  On the other hand,
the exact same thing could be said for any file format.  The name of the
game is communication, and anything that improves communication, including
a widely-understood file format, would be a Good Thing.

People have high hopes that XML will indeed become such a lingua franca.  I
agree that it would be a great thing if it did.  However, it hasn't yet,
and it won't at any time in the near future, and anything beyond the near
future is in the realm of guesswork.

Jonathan Brecher
CambridgeSoft Corporation
[email-masked]

---

From: "Wolf-Dietrich Ihlenfeldt" <[email-masked]>
Date: Tue, 18 May 1999 22:49:23 -0600

> A lot of people do, myself included.  Does MS Excel read xml files?  No,

It does not read text files in general, only tab-separated (or
similarily structured) table data files, if you want
a sheet you can reasonably work with. It does not read the
atomic charges from a MOPAC result file, or a series
of such files. You have to
prepare the output by going through the result file and extract
it into an intermediate file, manually,
if you do not have a converter script which needs constant
maintenance. And a simple
tab-sep table is certainly not useful as a general output format.
Office 2000 will presumably parse an XML-file of any origin and
let you graphically pick the data columns which should
be imported.

> lack of connectivity tables, at least in the programs i use.

a) PDB can encode connectivity (though not bond orders)
(No, I am not defending PDB - I think PDB should also
be abolished in favor of a more flexble approach ASAP)
b) How do you encode connectivity in your 'text-file' tables?
And how do you obtain this information from, say, Gaussian?
What kind of format do you use? To me, it is not yet
clear how do you think an output should look like. Your
notion of a 'text file'  confuses me.

---

From: Richard Wood <[email-masked]>
Date: Tue, 18 May 1999 16:02:36 +36000

I still say why convert many file formats to one if you don't have to/not 
all applications curently being used will handle them.
  
It's like assuming that everyone using a PC is using Windows 95/98, when 
the fact is that some are still using Windows 3.1.

Richard

---

From: "Rzepa, Henry" <[email-masked]>
Date: Tue, 18 May 1999 23:48:56 +0100

>...XML would be appealing if it required that well-formatted
>HTML does.  ...All arguments I've seen for XML

A full answer to  Jonathan's point is deserved, but for the time being I
will confine myself to saying that comparing HTML and XML
is unfair. The comparison should really be with XML and SGML
(which is widely regarded in document publishing as a great success
and has been around for  15 years).
A HTML server accepts any document, well formed or not;  an XML repository 
will accept only  a well formed (ie balanced tags)
and valid (ie conforming to a specified  DTD)
XML document; Arguably, an XML
repository is closer to an object-oriented database than a HTML
server., but with style sheets there is no reason it cannot be more
powerful than HTML.   XML 
IS much better formed and structured than  HTML (HTML itself is
migrating to  XHTML in recognition of this)
and I think Jonathan is wrong about  XML
not making it in the near future. The last 3 months have seen a
remarkable uptake, although not obvious to the public at large.
See http://www.lists.ic.ac.uk/hypermail/xml-dev/
for examples.

Although much  XML development
is taking place in  commercial environments, we should encourage
OpenSource thinking in an area such as chemistry, since it is only
by such  collaborative efforts that it will succeed in our community.

I might also add that a lamentably unlauded
project called HyperWave (originally  HyperG) solved many of the problems
of "bad" HTML and broken servers by only allowing well formed 
marked up documents to be entered and served.
It also handled links correctly, as well as metadata, queries, and many other
things now considered good practice! It was introduced ONE year
before Tim Berners-Lee invented  the Web at CERN, and had IT
caught on instead, things might have been a little different! 
It was literally ahead of its time, and the  World was not ready for it.

Henry Rzepa. +44 171 594 5774 (Office) +44 171 594 5804 (Fax)
http://www.ch.ic.ac.uk/rzepa/