Explorations in software troubleshooting
I was running a performance test recently, and I saw some odd behavior. My usual process is to restart the application, let it finish initializing, and then run a simulation for about an hour. That’s usually enough time to collect enough samples to get a good idea of the performance of various requests. My 1 …
I was recently reminded about the difficulty in seeing what’s there by my coffee machine. My mental model of the coffee machine is pretty simple: I put in the grounds, I add water, and I press a button to make coffee. My machine has some other features and some other buttons, but I don’t care …
When I go to a movie, I sometimes sit though the credits (especially when it’s a Marvel movie!). It is remarkable just how many people work behind the scenes to bring a movie to the screen. You can stretch that analogy a bit to a software system in that there are often many things happening …
History never repeats itself but it rhymes. Attributed to Mark Twain. It is unusual for a problem to only happen once. But one of the maddening things about complex problems is that they are not often reproducible on demand. They are reproducible in a random, rhyming way. All we usually have to go on is …
Marty, you’re not thinking fourth-dimensionally! Doc Brown, Back to the Future III It is easy to underestimate the power of time. In fact, one of the big differences between a test system in the lab and a deployed system is time. As time passes, the number of people using a system can grow, along with …
“The first rule of good puzzling – see things as they are, and not as they seem.” Elizabeth Haydon, The Floating Island. Perhaps the biggest challenge for a trouble hacker is to see what’s there. Sometimes when you are observing a system, you’ll only see what you expect to see. If something happens that you …
Let’s look at how a mental model can evolve during the problem-solving process. Let’s say you have two separate applications, deployed onto two separate servers. These applications don’t interact with each other, so your initial model might be something like this: This model could be useful for some things. You might reasonably expect that you …
Since mental models are so important to trouble hacking, it seems only appropriate to start with this observation from mathematician George Box. All models are wrong; some models are useful. Box, G. E. P. (1979), “Robustness in the strategy of scientific model building”, in Launer, R. L.; Wilkinson, G. N., Robustness in Statistics, Academic Press, pp. 201–236. …