I'm calling this "Science for Busy People", but maybe it's not actually science. I'm not a scientist, and have never trained as one.
This stream is based on three assumptions:
The intended audience is working programmers.
I'm going to start with some introductory material—a parable, an explanation of that parable, defining some terms—and then I'm going to get into some specific things that I think can help.
Say we're going to write a new computer program. Programmer Zero says we should write it in Rust, Programmer One says we should write it in Haskell. Programmer Two says we should be scientific about this.
Zero: Let's write our new web app in Rust!
One: No, let's write it in Haskell, it's much more mature.
Zero: Rust has much better tooling and documentation, and is has a gentler learning curve.
One: Haskell's awesome type system will prevent bugs before they happen. Besides, we don't need a systems programming language for a task like this.
Zero: Yeah, but nobody uses Haskell. Rust, on the other hand, was recently used for $AWESOME_THING and they said their experience of it was great and they just IPOd for five billion dollars.
Two: We shouldn't make such important decisions lightly. Let us be scientific about this.
Zero: OK, let's do an experiment. I'll spend a month writing it in Rust, while at the same time One can spend a month writing it in Haskell. At the end of the month, we'll see which is better.
Two: But how will we decide which is better?
One: Well, I guess which one is most feature complete, and performs the best.
Two: Performs the best how?
One: Since it's a user-facing app, I guess latency is the best metric?
Two: Maximum latency? Latency for all requests? Latency under what load? With how many servers? Would the Go one be better if it had lower maximum latency but required twice as many servers?
Zero: Yeah, also, we haven't mentioned bugs. Or maintainability. I don't even know how to measure that.
One: And I've got to do twelve interviews over the next month, and I'm on holiday next week. Maybe we'd get to the end of it and decide that Rust was better but actually it was just because I just couldn't do as much work.
Two: So we can't make a decent decision about which is better. Therefore, we should do it in Python, which is what we've used for everything else.
Zero: That doesn't make any sense.
(enter Manager)
Manager: I just went to a conference and everyone was talking about Go. Let's write the new thing in Go—I want to see it live in two weeks. Also, can we make it a microservice?
Initially, Zero & One are talking about heuristics for measuring how good something is: maturity, quality of tools, documentation. Then Zero tells an anecdote about someone who agrees with them.
Two rightly points out that this is a terrible decision making process, and then Zero and One start hashing out a way of doing an experiment. They quickly realize that there are way too many variables, and that nailing down a definition of "good" or "better" that can actually be experimentally verified is essentially impossible. All the things they can measure are incredibly specific, and they've got no way of mapping back from there to a general notion of "goodness".
So, what are we to do?
Like Two, we could use the difficulty of making rational, scientific decisions as a reason to stick with the status quo. Or like Manager, we could look to a trusted authority: the crowd; our instincts; a trusted advisor.
Can we do better? Is there a way we can actually use science?
As I said, I'm not a scientist, so I'm going to pull in a few ideas from Richard Feynman, who was a scientist, and whom a lot of people say was a pretty good one.
The test of all knowledge is experiment
I like that. I don't think it's entirely true, but if you're going to do science or be scientific, then you probably need to think that way.
Science is more than just doing a lot of experiments, it's also:
And probably many more things. All of this is a lot of work.
Before we can get on with proper sciencing, we're going to need to check our equipment.
Our primary equipment are brains. We each have direct access to one, and indirect access to many more.
I don't know what brains are made out of, but they are pretty useful. They also don't behave the way you might think.
Daniel Kahneman has written an amazing book called Thinking, Fast and Slow that goes into this in detail, and really ought to be read by everyone. You can also read about cognitive biases on Wikipedia.
In a nutshell, the first and most important thing we can do to be more scientific is to distrust our brains. Our guesses are probably wrong, our memory is probably wrong, the examples that we think of are suspect, the estimates we make are overly optimistic, we value stories over data, we find it hard to process information that doesn't fit with our existing opinions, we answer the questions we find easy rather than the questions that matter.
Next time: our next-most important tool, and some ways we can use it.