History never repeats itself but it rhymes.
Attributed to Mark Twain.
It is unusual for a problem to only happen once. But one of the maddening things about complex problems is that they are not often reproducible on demand. They are reproducible in a random, rhyming way.
All we usually have to go on is what has happened in the past. And when we make predictions about the future based on past behavior, we are working from a mental model which assumes the future is like the past. This certainly seems like a reasonable model, but in the spirit of all models are wrong, let’s see how this can go wrong.
It starts with problems in our observations.
Extrapolation
If we observe a system over a particular time period, then if the future is like the past, we can extrapolate our observations into the future. This extrapolation assumes that the system behavior is stable and that the time period we observed is representative of all future time periods. This is often not true, and so the further we look into the future, the less accurate predictions are likely to be.
Here’s an example taken from a performance test I conducted. Here there are three different kinds of behavior in three different time frames. If I observed the system only at those times, I would make very different predictions. In region A, there is stable performance and so you’d predict stable performance in the future. In region B, the performance is degrading steadily. If you only looked at region B, you would predict ongoing degradation. In region C, performance is varying greatly so you may not feel comfortable with making any predictions at all if that’s all you had to go on!
Interpolation
If we make a series of observations over time at some rate, we will usually assume that the behavior of the system between observations was similar what we observed. We interpolate the observations to cover the times when we weren’t observing.
So what are we likely to miss? There is some insight in the field of digital signal processing – the Nyquist-Shannon sampling theorem. This theorem says that you need to sample at twice the rate of the highest frequency in your signal. Applying this to our software system means that we can only detect patterns that happen less often than half our observation rate. In other words, if I take observations every hour, then I can detect patterns that occur every two hours but not any faster than that. A behavior that occurred every 30 minutes would be lost in the interpolation.
Asymmetry
There’s another assumption that we can make if the future is like the past. We can also expect that behaviors are time-reversible. For example, if you were measuring system performance as a function of user load, you’d expect that the measurements would not be different if the load were increasing or decreasing. A measurement at a load of 50 users should not care whether the last measurement was at 30 or 60 users.
Systems are not always time-reversible. One example is hysteresis, where the state of a system depends on its history. One form of hysteresis in a software system would be when a bottleneck forms around a critical system resource. If the bottleneck takes time to dissipate, then the system will perform better as user load increases and will be worse as user load decreases (until the bottleneck dissipates).
Aggregation
One form of asymmetry that is worth calling out separately concerns aggregation of data. One example of aggregation would be taking the average of a set of measurements. I use averages quite a bit in analyzing performance tests, but with care, because asymmetry in sample values can distort the average.
One extreme example might be a web service which usually runs in 2 milliseconds, but occasionally takes as long as 100 seconds. If there were 99 samples of 2 milliseconds and 1 sample of 100 s, the average is just over 1 second. If I used this average to predict the values of future measurements, I wouldn’t even be close. My predictions would either be way over or way under the actual values.
It’s a good idea to use the average in conjunction with other aggregate measures like standard deviation, range, or percentile values (like the 90th percentile). These other numbers can put the average into perspective and warn you about potential asymmetries.
Conclusion
The key takeaway from this discussion is that the future is sometimes like the past, but this view is just another mental model. Because it is a model, it is wrong. But it can still be useful!