Explorations in software troubleshooting
When I go to a movie, I sometimes sit though the credits (especially when it’s a Marvel movie!). It is remarkable just how many people work behind the scenes to bring a movie to the screen. You can stretch that analogy a bit to a software system in that there are often many things happening …
History never repeats itself but it rhymes. Attributed to Mark Twain. It is unusual for a problem to only happen once. But one of the maddening things about complex problems is that they are not often reproducible on demand. They are reproducible in a random, rhyming way. All we usually have to go on is …
Marty, you’re not thinking fourth-dimensionally! Doc Brown, Back to the Future III It is easy to underestimate the power of time. In fact, one of the big differences between a test system in the lab and a deployed system is time. As time passes, the number of people using a system can grow, along with …
When a web site goes down, the trouble hackers tasked with bringing it back up are under tremendous pressure to act. When the pressure is high enough, you may have to take actions before you really understand the problem. Sometimes these actions uncover new information that leads to a solution, but sometimes, they just add …
“The first rule of good puzzling – see things as they are, and not as they seem.” Elizabeth Haydon, The Floating Island. Perhaps the biggest challenge for a trouble hacker is to see what’s there. Sometimes when you are observing a system, you’ll only see what you expect to see. If something happens that you …
Let’s look at how a mental model can evolve during the problem-solving process. Let’s say you have two separate applications, deployed onto two separate servers. These applications don’t interact with each other, so your initial model might be something like this: This model could be useful for some things. You might reasonably expect that you …
Since mental models are so important to trouble hacking, it seems only appropriate to start with this observation from mathematician George Box. All models are wrong; some models are useful. Box, G. E. P. (1979), “Robustness in the strategy of scientific model building”, in Launer, R. L.; Wilkinson, G. N., Robustness in Statistics, Academic Press, pp. 201–236. …
When I think about the practices that have helped me the most in hacking trouble, I find that they fall into three broad categories. These three categories form the troublehacker’s triangle: Let’s start at the top of the triangle, with mental models. When I reflect on problems I’ve solved in the past, I distinctly recall …
This is a blog about broken things. After all, most of the software systems I work with are broken. To be fair, I’m usually the one breaking them so I’m not complaining. When I work as a system or performance tester, that’s part of the job description. You’ve got to push the limits of the …