Many senior members of the academic community rely on their reputation as researchers to brush aside basic issues with the software that they develop for scientific purposes. These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability. The lack of focus on research software quality among senior PIs, funding committees, and article reviewers is a huge problem in academia. The problems with the Ferguson model and codebase are just the latest prominent example of this.
The tools that academics have access to are excellent. If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. I think the OP is wrong to suggest otherwise.
The problems with the Ferguson model are an opportunity to educate more members of the community about the fact that good software development practices are not optional for good science, and that senior members of the community jeopardize their own reputation by not paying attention to them.
> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability.
Or maybe these things aren't actually as important as we think they are in professional software development?
If they're able to produce useful scientific results (in general, not specifically in this case) without those things then maybe they don't matter as much as we think they do?
These qualities are important for exactly the same reasons they are in production: Without those qualities, your code is brittle, your deploys are brittle, changes are brittle.
It's just like saying "It runs on my machine". The scientific term for this is "Replication crisis [0]"
Clearly not, but that's because what they need to achieve is publication of something interesting, not something correct.
Academics have a lot of excuses for writing terrible code. It's really shocking. So far I've seen:
- It doesn't matter if the code is buggy because the results don't need to be accurate
- We can't afford to write good code.
- We aren't properly trained to write good code.
- We just average the results and that fixes the bugs (!)
- We aren't computer scientists so what do you expect.
- Science is special which means we don't need to write tests to get correct results. We just eyeball them and know if they're right (so why bother writing a model at all then?)
- You can't criticise or judge us because mere software developers can't understand science.
- Nobody told us it's easy to screw up memory management in C
And a whole lot of other baffling and insulting nonsense. How many of these excuses would be accepted if a private company produced a dangerous product due to bad code, and produced this litany of BS in a courtroom? None.
They aren't idiots and they aren't ignorant of how professional software developers work.
It's apparent from some of the responses to this fiasco that they are totally ignorant of how professional software developers work. And they're proud of it, which in my view makes them idiots too. You can't both tell governments and whole societies to "follow the science" and then blow off any suggestion of working to professional standards.
> Are you sure that writing tests makes science better? Or are you just assuming that?
It is perfectly acceptable to write code without tests. Proof-of-Concept or Minimum Viable Product are a great place to write code without tests.
It is less acceptable if other people will run or use that code. It is even less acceptable if anyone (including oneself) ever updates or extends the code.
---
You could take this analogy to scientific instruments. Imagine you make a novel particle detector. You get a scientific result with your detector.
A colleague uses your detector, but they don't clean it properly before use, and they use a power supply with lower voltage. They don't detect any particles! Was your science bad? Would the "science be better" if there were clearer instructions and pre-requisites?
Now imagine another scientist makes a copy of your detector from the description in your paper. They get some stuff wrong because your description was ambiguous. Was the science bad?
---
By the way, all of these things are real problems with scientific investigations, and not just in the software realm.
If the results aren't reproducible, they can't be assumed to be true. Then they're only useful if you only care about publication and not about whether the results are actually true.
> If the results aren't reproducible, they can't be assumed to be true.
I don't understand why having brittle code would mean that the results are not reproducible? You don't need to modify the program to do a reproducibility study.
Reproducibility is one of the issues mentioned earlier in the thread. And being able to audit the code and understand what it actually does seems rather important too.
Move fast and break things has its downsides: work that influenced public policy/political debate was retracted after an error in the model was discovered [1], errors have been found in widely cited databases [2]. These are things that should be avoided I'm sure you'll agree.
Science has this additional problem that its memory is short - mistakes seem to be discovered when work is in the long tail of the citation curve, once it's out of the news. Even if you retract a paper, there is no easy way to trace the contagion to the work that uses it. That's before you consider mistakes that might be deliberate [3].
I have no doubts that more software/data rigour would make science more accurate, but the cost would be substantial, and it would no doubt slow down discovery until the benefit of open source kicked in.
> Or maybe these things aren't actually as important as we think they are in professional software development?
There's something to this. Most academic software is a simple one-off development, with little consideration for long-term maintenance. OTOH it's not likely that undocumented, untested and unfixable software can produce actual "useful scientific results".
Do you need tests to do that? Are tests the only way?
Are you doing formal verification of your software? Why not? If you're not bothering to do that why are you criticising researchers for not bothering to use tests?
This is, indeed, the whole truth and nothing but the truth. My own anecdote: through happenstance I'm co-author on a paper, a large number of my friends are PIs/PhDs etc and one of them needed help after the software written for them by the CS student they'd been assigned didn't work (he got a masters for it, I can't believe anyone in the CS department looked at his work). I asked about publishing my code after the paper was published and no one seemed bothered.
I've also seen my friends on the mill of, shall we call it paper chasing? That seems most apt. It reminded me of borrowing someone's homework and making minor changes to get it past teacher - except the homework was the same paper being constantly rejigged for submission to a new journal. Not much truly new work seemed to be going on.
I couldn't agree more. We're in the middle of a reproducibility crisis. It's extremely important that researchers start putting more effort into quality research instead of quantity. "Publish or Perish" is killing academia.
> These include the lack of testability, debuggability, reproducibility, separation of concerns, documentation, or usability
Have you actually checked to what extent the Ferguson codebase is guilty of these things? It seems to me there are people spreading misinformation, and too many people taking their word for it.
Academics are not software people, by and large. The code is custom built for a specific graph or statistical run. Typically the people writing the software are grad students with little to no interest in code; they just want the answer. Their interest is in the thing they are studying, not code. They may not even have coded before grad school, nor know matrix algebra or calculus. Excel is a big step for them.
I have experience in this issue. A fellow grad student of mine asked for some help with his code. I said sure, but it'll cost you a 12 pack of beer. After the 11th nested 'if' statement in Matlab, I upped it to a 24 pack. We never did get it working right.
> If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great.
These courses are usually short introductions into python or Java. Most scientists simply don't have time in their curriculum for a full software engineering course.
I didn't claim they were good. My father is doing a Python one right now and I'm not very impressed. He is retired from the University but is still eligible to take the course.
That's the whole problem with the "Learn to code" movement, tbh. Plus Python and Java are terrible languages when it comes to those broader swdev/softeng concerns.
The tools that academics have access to are excellent. If you compare the free support academic software developers receive from the rest of their community to other engineering disciplines, it's beyond great. I think the OP is wrong to suggest otherwise.
The problems with the Ferguson model are an opportunity to educate more members of the community about the fact that good software development practices are not optional for good science, and that senior members of the community jeopardize their own reputation by not paying attention to them.