I lean towards more open code and data, but could one argue that if the code uses some fundamentally new, non-trivial methodology, a replication of the research should write new code implementing that method from it's description. Otherwise, if the original code is re-used, a bug in the code may very well cause the replication of an erroneous result.
There are two reasons to look at the code. The first is to look for bugs. Obviously that is easier if you have the code. Testing is much easier than writing from scratch. The second is to see how robust the results are to changes in assumptions and other aspects of the methodology. When you are talking about a big computational problem, it is not realistic to expect others to do a full, independent replication. And if you can't reproduce the original results, you need access to the original code anyway, to find the source of the differences.
In fact this is the main reason why I don't subscribe to the idea of research papers including code (as a requirement). Independent replication should rely on independent implementation. Unless the software is too generic, the original researcher should have the advantage of using the implemented software for new discoveries, as much as a traditional scientist has the advantage of using his own lab for additional research.
Be able to reimplement the code < That's an interesting perspective. However I think the vast majority of simulation work never gets that sort of scrutiny, no matter if the code is open source
Personally though I think we should give up on papers and everything should be published on an ipython notebook style page where others can play with data and code.
Conversely, if the original code isn't available, the community may never know why they can't replicate the results. This could especially be a problem if the original researcher is more known and respected than those attempting replications (i.e. devolving into he said she said).
Sure thing, the best case is where the original code & data is available, and a cross check replication does an independent reimplementation. There's a little bit of risk that peeking at the original code might lead one down a wrong path, but overall I think it's still better that the original code is provided. For example, if someone isn't looking to replicate and might be instead trying to extending the original work, using the original code could save a lot of time (and could help find original mistakes too).
