From owner-chemistry@ccl.net Tue Oct 18 12:49:00 2011 From: "N. Sukumar nagams(!)rpi.edu" To: CCL Subject: CCL:G: Science code manifesto Message-Id: <-45679-111018115633-3430-3UE551MwkRUeIjB+TLQleQ~~server.ccl.net> X-Original-From: "N. Sukumar" Content-Disposition: inline Content-Transfer-Encoding: binary Content-Type: text/plain Date: Tue, 18 Oct 2011 11:57:09 -0400 MIME-Version: 1.0 Sent to CCL by: "N. Sukumar" [nagams**rpi.edu] Economic and national security concerns could be legitimate issues, but hiding science from the public (who pays for it) in order to protect it > from luddites or from political attacks is the most ridiculous thing I've heard yet. They used to do something like that in the Dark Ages, hiding whatever science there was behind magic spells, rites of initiation and oaths of secrecy - and we know how well science flourished during those times. They were called the Dark Ages for a reason! At best such tactics will be counter-productive and discredit such secret scientific results. It would then truly be a case of My (Holy) Book versus Your Book! As I read it, the manifesto only applies to results meant for publication. Since classified defense research does not generally make it into publication, the issue would therefore be moot with regard to genuine national security concerns. And yes, I can imagine Gaussian making their code available for public scrutiny. In fact, they do so now; many scientists around the world have access to and are able to scrutinize the source code, or even modify it for their private use. Their restrictions concern reproduction (copyright), redistribution, re-engineering and publication of benchmarks related to performance, not whether there are bugs in the algorithm or not. It was also mentioned in this discussion that journal editors/reviewers do not have the time or resources to verify source code or reproduce results and that you cannot expect a research group to shut down their lab while you try to reproduce their results on their systems. By and large this is generally true, but in cases of exceptional interest/importance (e.g. the case of homeopathy), Nature for instance has been known to send out a team of editors/reviewers to do just that. However, it is unrealistic to expect unpaid reviewers to bear the full responsibility for fact-checking all the intricate details of the code used to generate scientific results. On the whole, this works best by the self-correcting, re-engineering, crowd-sourcing nature of the scientific enterprise, but having the raw data and source code available for inspection in those exceptional circumstances (as in the example above or as in the case of anthropocentric global warming) is important for ensuring transparency and confidence in the scientific results. Dr. N. Sukumar Rensselaer Exploratory Center for Cheminformatics Research http://reccr.chem.rpi.edu/ -------------------------- "A Drug is any substance which, when injected into a rat, produces a publishable, scientific paper." ==============Original message text=============== On Tue, 18 Oct 2011 8:51:41 EDT "Mark Zottola mzottola ~ gmail.com" wrote: "Although politics may be important in climate modeling, I thought that this was the computer chemistry forum." Monte Python once did a skit that you never know when the Spanish Inquisition will show up. I used weather modeling as an example, but the bigger point is that who knows what ox is next to be gored. Computational chemistry studies (QM, MM/MD, QMD etc.) on methane clathrates could be the next battleground the luddites in the US would latch on to. Or There seems to be a yin and a yang to this idea. Full disclosure of everything to anyone sounds good. But there are business concerns to be addressed - can you imagine Gaussian putting their code in for full public scrutiny? Government scientists using home grown codes face publication issues due to overzealous Public Affairs Officers or their superiors refusing to release codes in the vague name of national security. Similarly, in closed systems like China who may refuse publication of their internally developed codes, would we lose significant knowledge because we refused to accept their papers due to this "manifesto"? Finally, the specter of Mencken's booboise misusing that access to information. On the other hand, the QCPE was a great idea, but it seems to have died in the academic rush to make money. Having codes readily available for dissemination and use would be a positive for the community. Total non disclosure, copyrighting and hiding codes does seem to protect the economic interests of those who have excelled in making money off their codes. It sanctifies the current economic model used for making money in computational chemistry. While there are benefits to this mixture of economics and secrecy, the question is whether this is consistent with the goals of science. The current two-tiered system in place - some software available for public scrutiny, others protected as though it were the secret to the universe probably does need some tweaking. But it is hard to see how a far-ranging manifesto like this is going to solve much considering the utter disruption its full implementation would cause. On Tue, Oct 18, 2011 at 1:42 AM, David A Mannock dmannock[a]ualberta.ca < owner-chemistry###ccl.net> wrote: > Although politics may be important in climate modeling, I thought that this > was the computer chemistry forum. OK, so some chemistry happens in the > atmosphere, but accurate simulations of those conditions are also important. > Whether a scientist or a member of the public finds an error in such code > may not be important, although it does not help when people make mistakes. > Science works iteratively, through criticism from all sides. Where I see a > problem politically is where people (especially those in science) have > models that they have spent years working on and whose funding is dependent > on those models. Not disclosing experimental protocols/computer chemistry > codes maintains secrecy and it becomes more difficult to break down a poor > model. I have fought this in science for 30 years and it is hard work. We > are all dependent on peoples honesty and integrity, and if this info is > disclosed when the manuscript is submitted for publication, bad code or > unclear justification of the code can be parsed by the reviewers and > clarification requested. It does not help where there is a conflict of > interest between the reviewer and the authors of the MS, but that is the > Editors job. > > It is easy to make an argument for doing nothing, governments and > universities across the world have raised this response to an art form. Is > this something that we should accept and be happy about? > > > On Mon, Oct 17, 2011 at 7:09 PM, Mark Zottola mzottola{=}gmail.com < > owner-chemistry*ccl.net> wrote: > >> For all our concerns about the impact of this manifesto on science, I >> think most are missing what I believe to be an important aspect. If a code >> is readily available for the scientific public, then it is readily available >> to the general public. As one area mentioned in the manifesto was climate >> modeling, it is clear that enterprising non-scientists or muck-raking types >> can access these codes. >> >> In any well-commented code, there are bound to comments like "applying the >> Fandoozie trick" or other such innocucous statements. In addition, in any >> code of over 1000 lines there are bound to be bugs. Can you imagine the >> public discourse when some rabel-rouser claims the software for climate >> modeling is flawed and he shows the offending code as "proof". Flaws and >> "tricks", common to all software, have the potential to be exploited for >> political gain under this manifesto. >> >> I'm not sure where the middle ground is when trying to make software open, >> accessible, and maintaining peer review without the introduction of >> politics. I think this needs a lot more thought and consideration. >> >> Mark Zottola >> > > ===========End of original message text===========