When I go to a movie, I sometimes sit though the credits (especially when it’s a Marvel movie!). It is remarkable just how many people work behind the scenes to bring a movie to the screen.
You can stretch that analogy a bit to a software system in that there are often many things happening behind the scenes. There are the obvious things like backups. But there can also be threads running in the background carrying out maintenance activities.
Some threads might run to clean up cached data. In a data warehouse, background jobs aggregate data into daily, weekly, or monthly tables. Many of these activities happen overnight, when there is less load on a system.
Are you feeling a disturbance in the Force? This discussion should be enough to make your spider sense tingle, if you’re a troublehacker. System behaviors that are invisible are hard to incorporate into your mental model.
Let’s look a bit deeper at one example: statistics generation in relational databases. I was recently looking at a performance problem which arose rather suddenly. Nothing out of the ordinary had happened recently – one day, the system was working fine and the next day, it was slow.
I eventually narrowed the problem down to a set of slow database queries. Those queries normally run in a few milliseconds, but they were now taking 30 seconds. What had changed?
The answer: the database had updated its statistics overnight, and this led to the selection of new execution plans for the queries. These new execution plans were not optimal. This is a common problem with databases – you can Google “SQL plan stability” to read more.
The main point here is that a sudden change in system behavior can be a sign of invisible, background activity. If your mental model doesn’t cover that possibility, it is time to evolve your model.