CCL: Science code manifesto

 Sent to CCL by: Konrad Hinsen [hinsen||cnrs-orleans.fr]
 On 16 oct. 11, at 11:05, Sebastian Kozuch kozuchs^_^yahoo.com wrote:
Let’s say that I receive a manuscript for review that uses program X, which I may have access to its code now (since I’m the reviewer). Do I have to spend a week trying to understand its algorithms to check if the frequencies are correctly calculated? For me, most (if not all) the programs are in practice black boxes, as I’m hardly a programmer. Therefore, open or closed software (usually) doesn’t make me any difference, except from a philosophical perspective.
It may not make a difference in the routine operations of science, but in case of exceptional importance of some work, it makes all the difference. It is then up to the community to decide in each case whether the effort of finding the right expert for such an evaluation is worth the trouble.
Now, let me consider an analogy from experimental chemistry. I have to review a manuscript where the authors tested some compound with a 600 MHz NMR. My lab is much more humble, and I only have access to a 400 MHz NMR. Id est, I cannot reproduce the results of the paper that I’m reviewing. Is here any difference compared to the theoretical case?
Yes: you can always find a 600 MHz NMR spectrometer if you think it's worth it. You can also hand back the review to the editor of the journal and say it needs to be done by someone with the right equipment.
I had in a couple of cases problems to reproduce the results of papers, mostly because the authors didn’t provide enough information. However, for 100% reproducibility I need 100% the same conditions (and a lot of resources and time). This makes the selection of the software just a small issue.
It is just one issue, but it's one that we can address, so why shouldn't we? Insisting on sufficient descriptions for reproducibility is another action we can take right now, so we should as well. Personally I do in all reviews I do, and all authors have complied so far.
I would like to hear the thoughts of other people about this issue, as I consider the peer reviewing process a very complex and far from perfect system.
I agree, but my conclusion is that we should take every possible step to improve it, even if it's only a small step. 
 --
 ---------------------------------------------------------------------
 Konrad Hinsen
 Centre de Biophysique Moléculaire, CNRS Orléans
 Synchrotron Soleil - Division Expériences
 Saint Aubin - BP 48
 91192 Gif sur Yvette Cedex, France
 Tel. +33-1 69 35 97 15
 E-Mail: research at  khinsen dot fastmail dot net
 ---------------------------------------------------------------------