Subject: Validation of Modeling Software

From: [email-masked] (Hopfinger at UICVM)
Date: Tue, 20 Sep 1994 13:45:06 -0400

Validation of Modeling Software

1. Background

        With the current high level of interest in molecular modeling,
increasingly sophisticated software is being offered by over 200 commercial
organizations in the US. and Europe. This software can be quite expensive
and often requires advanced computer platforms for its successful use.
Above all however, it is complicated. A typical modeling package may
contain some ten program areas, each of which has numerous working levels
and at each level there may be many commands, each of which offers many
options to the user.

        Overall such a package may therefore have hundreds, or even
thousands of commands and as a practical matter, users often restrict their
activities to one or two specific program areas. They have less interest
and little experience with the other available areas, but in the areas with
which they work, they gain considerable experience and soon become experts
in the particular subject.

        The accuracy, or validity of the results that can be developed by
modeling programs is usually weighed by experts who use an informal process
of external validation, i.e. comparing new result to results obtained in
some other way. This may be merely use of an earlier version of the program
or some truly external method, such as x-ray crystallography.

        Except for work done by individual users of the software,
systematic efforts to provide external validation of modeling software have
not been widespread. Validation carried out by individual users has a
number of disadvantages. It tends to be focused upon the specific problem
of interest, and is usually not published widely. More seriously, it can be
significantly time-consuming.  A small informal poll that we carried out in
May/June, 1994 suggested that molecular modeling groups spend from
person-months to well over a person-year working on the validation of new
versions of familiar modeling software. This is wasteful of highly trained
resources and moreover, because it has a narrow base, it is a validation
scheme with little robustness.

        We propose here a more broadly based validation scheme, which will
cover many of the competing computer programs and which, ideally, will be
carried out by an independent group, which can retain an objective approach
to the problem. Much in this proposal requires final definition and the
proposal as a whole must be acceptable to the community if it is to
succeed. For these reasons, a meeting to discuss the issues is being held
in Baltimore in November, 1994. All interested parties are invited to
attend and conclusions arrived at by consensus will be published in early
1995 in the Journal of Chemical Information and Computer Sciences. The
purpose of this message is to present a tentative agenda for this meeting
and to invite you to attend. If you feel your organization or company has
been overlooked in this draft agenda, please contact us to be included in
the endeavor.

2. Problems to be addressed

        Molecular modeling consists of a relatively large number of
operations, many of which have their own character and corresponding
software. It is not realistic to attempt to cover all components of
molecular modeling in a first-pass analysis of validation of modeling
software. Rather, acknowledging the breadth of the undertaking, and
focusing on the current major molecular modeling components and tasks seems
the best strategy to pursue. One working list of priority validation topics
is,
*       Molecular mechanics     - force fields and parameters
*       Statistical methods     - QSAR/QSPR, data analysis
*       Conformational analysis
*       Docking and intermolecular Interactions
*       Molecular similarity analysis
*       Environmental / solvation methods
*       Molecular dynamics

3. Other Issues

        Very little is known about the process of validation of molecular
modeling systems, and a minimalist approach seems best. The criteria laid
out above represent a small first step towards a systematic validation
effort. Determination and comparison of the structures of small flexible
and rigid molecules are the most common of modeling tasks and offer
themselves as a basis for validation.

        Still, one must be cognizant of the growing variety of applications
of molecular modeling, and the corresponding added dimensions they bring to
the validation problem. There is a need to identify what validation goals
are common to all molecular modeling applications, and what goals are
specific, but also central, to particular applications. For example,
polymer molecular modeling will have different specific end-points than
those of small-molecule drug design.

        Ab initio and semi-empirical calculations are not considered in
this iteration. The entire issue of validation of such calculations, is a
difficult issue, separate from that of dealing with molecular mechanics and
molecular dynamics.

4. Acceptance of Standards

        Software developers are as aware as any one of the need for
standards and of the role that validation could play in this area. They are
sensitive however to cost-benefit considerations and any process that is
too onerous (read: costly) will be adopted only with reluctance. This is
another reason for a minimalist approach. If the modeling community has an
agreed-upon and promulgated list of structures which can be used in
validation, then individual vendors can work on members of this list,
publishing their results, or not. The marketplace will respond to this and
the vendors who
made the correct choice will be rewarded.

        In this scenario, validation will work somewhat like an ISO 9000
standard. It represents an internally developed, self-guarantee which is
not mandatory but which may well facilitate sales to discriminating users.

5. First Steps

        The first steps in this effort must be an effort to determine the
feelings within the modeling community about these different issues. The
small informal poll taken in May/June, 1994 made it clear that a majority
felt validation to be an important issue, which so far has been neglected.
In order to make a more detailed assessment of the community's feelings, we
are proposing a 1 1/2 day open meeting at which the issues can be discussed
and some plan of action developed.

        This meeting is to be held on Wednesday and Thursday, November 9
and 10, 1994, at the Holiday Inn at the Baltimore-Washington International
(BWI) airport.

        There will be a registration fee of $100 per person, payable to
Pool, Heller and Milne, Inc., 9520 Linden Avenue, Bethesda, MD, 20814
(Phone: 301-493-6595; FAX: 301-897-3487; Internet: [email-masked]).  A
corporate rate of $89 (plus tax) has been obtained at this hotel which will
also provide meeting facilities and lunch for attendees.  The BWI airport
was selected as a site because various airlines, notably Southwest, offer
very economical fares to BWI. Some support for this meeting has been
provided by vendors and users of molecular modeling software .

6. Draft Agenda

                       Workshop on CAMD Software Validation

                                November 9-10, 1994

                                        Agenda

November 9

9:00 - 9:45     Introduction, Overview, Focus and Goals

                                1. Areas of validation
                                2. Models for validation
                                3. Validation support
                                4. Promulgation of validation data

9:45 - 10:30    Force Field Validation

10:30 - 10:45   Break

10:45 - 11:30   Validation of Small Molecule Modeling

11:30 - 12:15   Validation of Macromolecule Modeling

12:15 - 1:30    Lunch

1:30 - 2:15     Validation in Drug Design

2:15 - 3:00     Validation in Polymer and Materials Science

3:00 - 3:15     Break

3:15 - 5:15     Roundtable Discussion of Validation     - Users' Perspective

                1.      One hour of short statements by Representatives from
                        Industry, Government and Academia who are Users and
not
                        Developers.

                2.      One hour of Open Discussion

Evening -               Free for informal discussions

November 10

9:00 - 10:30    Roundtable Discussion of Validation - Vendors' Perspective:

                1.      One hour of Short Statements by Representatives from
                        Vendor Companies.

                2.      One hour of Open Discussion

11:30 - 12:30   Open Discussion.

                1.      Formulation of Conclusions and Goals and Preparation of
                        a Final Statement which will be Appended to the

                        Symposium Report for Subsequent Publication.

Please address all technical questions and issues to A.J. Hopfinger,
University of Illinois at Chicago, Department of Medicinal Chemistry and
Pharmacognosy, (m/c 781), College of Pharmacy, 833 S. Wood Street, Chicago,
IL 60612-7231, Phone: (312) 996 4816, Fax: (312) 413 3479, email:
[email-masked]

Hitesh Patel
MMaDD & QSAR lab
Dept. of Med. Chem. & 'Cognosy
UIC

---

From: Rafael Iosef Najmanovich Szeinfeld {S <[email-masked]>
Date: Tue, 20 Sep 1994 18:34:12 -0300 (EST)

I think that a fundamental step in validating modeling softwares
(MSoft's) is to compare the output from different softwares any attempt to
compare with the reality. 
	Sincerly yours,
				Rafael.
*-----------------------------------------------------
* Rafael Iosef Najmanovich Szeinfeld
* Dept. Biochemistry         -Chemistry Institute
* Dept. Mathematical Physics -Physics Institute
* University of Sao Paulo
* E-MAIL : [email-masked]
*
* MAIL:    Depatartamento de Bioquimica - BLOCO 10 INF.
*          Universidade de Sao Paulo  
*          Av. Prof. Lineu Prestes 748
*          CEP 05508-900 Sao Paulo - SP - Brazil
*------------------------------------------------------