If I've learned anything in my career it is that no, ideas are not valuable. There are vastly more bad ideas than good ideas. What makes an idea valuable is validation. Papers aren't to present ideas: papers are to present ideas that have been validated. We proposed an idea, we went and ran some experiments or gathered data some other way, and we concluded the idea was valid (or not valid). The point of this discussion here is that ideas that require huge amounts of computer effort to validate are prone to bugs. The conclusions cannot be relied upon to be validated without having the software available so that it, too, can be validated.
Really? I've read a lot of SIGGraph papers, and sure, they didn't provide all the code. But you know that the code exists. And certainly for those there's a lot of trust. I think we're talking here about something different. Not "hey, you can use quaternions for animation" but "If you factor in hippopotamus usage, young adults experience 22% more flange per square doughnut, and we got all this data, and we ran it through 25 separate computer programs written in lolcode, and look, proof!"
> But there are so many valuable papers in CS that just presented an idea. If you ignored them you’d be ignorant of how to do 90% of modern engineering.
You are right. But that is why we are having this discussions, so we can improve situation.
Having even bad code (and corresponding data) available is always better than not. You can always just ignore it, and read the papers like today.
Honestly I am ok with just zip file of project directory that you have anyway, with hopefully list of versions of os, libs and programs used.
We could do a lot better than just a zip file, but that would be a nice start.
For some papers. Others make a claim about some statistically significant look at data that might not have any basis in reality because the code is wrong. A famous example being the R&R paper in economics where a second look at the showed massive mistakes in the excel document they were using, invalidating the central thesis. Unfortunately not before being used by the world bank for years as a metric for forcing austerity on countries.
Not if the code is wrong, and therefore the conclusion may be wrong. I'm no scientist, but I don't think the point of scientific papers is to get unfounded ideas out into the world.
I can list many major influential papers in computer science that described an idea and didn't really give any concrete code, where we're still using the idea today.
For example the paper on polymorphic inline caching, which is the key idea for the performance of many programming languages today, just described the idea, and didn't present any code. How was it evaluated? People sat and thought about it. Holds up today.
You can reason about an idea through other things than concrete code. Code is transient and incidental. Ideas persist.
I think you're talking past each other. Both are true under different circumstances. In some cases an abstract idea is the important takeaway. In other cases the central point of a paper is to present conclusions that were arrived at based on analysis of some dataset. If the code used to generate or analyze the dataset is wrong then conclusions based on it likely worthless.
A lot of times the idea is wrong (and thus not valuable), and that can’t be proven either way without the code and data. So an idea that depends on code without the code is less valuable.
We can only possibly gain from publishing the code, and lose by not publishing.
It's not like it takes a whole lot of time to just dump your code in a github repo once you're done and link it somewhere on the paper (if you wrote code at all while working on the paper).
Sometimes I did just want to run my own experiments with different datasets, and those algorithms aren't always trivial to implement :|
