In this article, performance specialist Steven Parker discusses performance strategies in the first of a series of posts about performance and scalability. These tips can hopefully help developers identify and avoid bad strategies while choosing better ones.
We all like to think the code we write will run fast and work well. When it doesn’t, we need to figure out why. I see many people approach performance badly because it hasn’t been a daily routine part of software development, and now a problem has surfaced. One such example, Brendan Gregg has done superbly citing the following joke:
One night a police officer sees a drunk searching the ground beneath a streetlight
and asks what he is looking for. The drunk says he has lost his keys. The police officer
can’t find them either and asks: “Are you sure you lost them here, under the
streetlight?” The drunk replies: “No, but this is where the light is best.”
Brendan calls this the Streetlight Anti-Method. Some examples might be, I have resource utilization panels in my cloud deployment. I will just look at them because I have a performance regression. While that is a key part of one type of performance analysis, if you applied this method, and couldn’t find anything, you need to move on. Another example might be, you just know the only code change that could have caused this, so you keep changing that patch writing it in different ways, but you’re not finding it. You need to look some place other than under the streetlight.
Other bad strategies
Another approach to avoid is the Random Change approach. With random change, please count the case where someone “just knows” a particular piece of code is at fault. My law about performance of one’s own code: You will never know where your own code spends most of its time until you measure it. I count myself with this rule also. While I like to think, as a performance expert, I am more likely to predict where my code spends most of its time. I, too, am wrong. Never assume, always measure!
As a result of people believing they know how fast their own code is, they sometimes choose the Blame Someone Else strategy. “My changes in this release can’t be responsible for this, it must be your code!”
Finally, the Ad Hoc Checklist is also a bad idea. Did you enable the compiler optimizer? Did the hardware configuration change? Did a database table get a new column? While these lists have some value, the chances it works is pretty random.
How to measure in a meaningful way
Rather than going for any of the above approaches, I recommend you apply science and prevent performance problems by employing some positive patterns.
- Build performance measurement into your production deployment. Production is the only place to know for certain what your users’ experience is. Cloud providers all have logging that can be used to create custom metrics for your specific application.
- Create benchmarks in integration testing which measures changes in performance build-by-build.
- Validate the accuracy of your benchmarks by comparing production against test results.
If you follow good habits, you can turn performance into science, and create a great reputation for your application and your work.
Hopefully you enjoyed this article, and if you want to read more about these methods, check out Brendan Gregg’s book “Systems Performance: Enterprise and the Cloud”.
About Steve:
Steve Parker works as an Application Architect at Avanade Norway and has many years’ experience working with software engineering, operating systems, software development and project management. Originally from the US, Steve worked many years for Apple, Sun Microsystems and Fujitsu before joining Avanade earlier this year.
