Getting a handle on software metrics
Tue 11 Aug 2020 | Michael Baldani
Traditional metrics play a key role in determining how well you’re moving software from the beginning of the process to the end, but they don’t tell the whole story
Is your software delivery function doing a good job? For many organisations, the answer may seem obvious. If you’re meeting the traditional metrics you’ve set for efficiency, then, by that set of measures, you’re probably doing fine.
But, looking deeper, are you delivering on your underlying goal – to make software a true differentiator for your company? Are you serving up the high quality applications users want when they want them in the formats they prefer? Are you removing the roadblocks to excellence and rallying the organisation toward a common goal of delivering the most impactful software possible? Is the software you’re producing generating actual value for the organisation?
The answers to this second set of questions are more nuanced. Traditional metrics play a key role in determining how well you’re moving software from the beginning of the process to the end, but they don’t tell the whole story. The real measure of effectiveness is how well your organisation connects traditional metrics to a more qualitative holistic view of success defined by a new, emerging concept called Software Delivery Management (SDM).
What is SDM
In short, SDM is a strategy that puts software delivery where it belongs – not as a cost centre to be managed but as a true core function of an organisation’s business. Everything relating to the delivery of software ties together, creating committed resources and an aligned set of processes that delivers value both to customers and back to the business.
That means that all functions with any influence over the ultimate success of a software deliverable collaborate seamlessly – product stakeholders, developers, engineers, marketers, salespeople and business leaders.
A common data model captures all the information from the toolchain and makes it shareable. It also allows you to leverage that data to get clear visibility into and across your development and delivery process. Having these metrics and analytics brings development teams and stakeholders together, helping them be more aligned and working more efficiently by sharing insights up, down and throughout the organisation.
Peter Drucker’s old saying about business management, “If you can’t measure it, you can’t improve it,” provides a useful baseline. SDM is all about continuous improvement – so gathering metrics is a good place to start.
Using Metrics as a Baseline
The most common metrics that software delivery organisations follow come from six years’ worth of surveys conducted by the DevOps Research and Assessments (DORA) team. The DORA metrics measure how successful a company is at performing key functions integral to the DevOps process – most notably are velocity and quality.
Deployment Frequency (DF) measures speed: Organisations that build faster deliver more features more often. That’s important. Another speed-related metric, Main Lead Time (MLT), determines how long it takes to make a change and get the system back to deployment. Change Failure Rate (CFR) and Mean Time to Recover (MTTR) measure quality and stability. Low CFRs indicate that the organisation is doing a good job limiting the percentage of code mistakes that require rework. A low MTTR means the team can bring an environment back up quickly, in seconds or minutes after a change.
To arrive at the end goal of SDM – delivering overall value – the first things organisations need to do is balance the speed and quality sides of the equation. If you’re deploying multiple features a day, and the features you deliver are full of bugs, that’s a problem. Or if you’re releasing features once a month, and they’re super high quality, that’s great, but you’re going to lose users because you’re not integrating features fast enough.
DORA surveys evaluate DevOps proficiencies, breaking organisations into Elite-, High-, Medium- and Low-Performer categories. Some organisations may be satisfied with being Medium performers – based on the resources they have, what they’re building, their customer needs and their goals for user engagement. But, if you’re a large global enterprise and you’re not pushing out high-quality software quickly, your customers will notice and your competitors will take advantage.
Once you’ve collected your foundational DORA metrics, you’ll want to determine how your development effort impacts your business from KPI (Key Performance Indicators), goals and objectives and revenue standpoints.
If, for instance, an organisation has three teams building features for a product, how much is each team spending on each part of the process? Once the features get deployed, how are they engaging with customers? What is the feedback on those features? And ultimately how is that development effort returning revenue (ROI) back to the business?
You’re not only looking at the value and efficiency of your product team, but also at what the product team is bringing back to the business in terms of revenue. This takes software out of the realm of being considered a function that operates in the basement and turns it into a core business process which affects how the business runs moving forward.
DORA metrics can serve as a bridge between the technical and tactical outcomes of software development and the outcomes that drive the business. You want to establish the logical chain of metrics all the way from file churn, frequency of code commands and age of pool request up to metrics such as mean time to recovery, and then be able to establish a chain all the way to quantify what the metrics of the business are.
Over time you learn to capture what you can and figure out what adds value and what impacts your ability to derive those KPIs, then you can focus on how you influence those KPIs.
Applying the SDM Model
The SDM model can take the guesswork out of the metrics process. Here are a few examples of how it can work.
By bringing in data from different toolsets and generating metrics, you’re able to truly understand how a feature or product is progressing. Breaking down projects by teams, or better yet, moving from project to product with dedicated teams, gives you the ability to measure how effective each team is.
If Team A is delivering faster but at a lower quality, and Team B’s delivering slightly slower but the quality is a lot higher, you can gain insights that will benefit the overall process. You can leverage what Team B is doing from a quality standpoint and slow Team A down to match the level of quality.
From an efficiency perspective, having data relating to stalled pull requests and the number of pull requests gives you the ability to set policies that will drive further improvement. If a pull request has to be reviewed in 24 hours, and you have a policy around it to trigger a warning that it needs reviewed before the time is up, you know where to spend your time to alleviate that bottleneck. This helps you create guidelines to make sure reviews get done in a timely fashion, resulting in increased efficiency in your software delivery process.
Matching user engagement metrics for a particular feature with the costs it takes to develop that feature gives you a better look at the high-level ROI for a certain product. Your goal extends beyond just speed and quality: you want to measure the impact on the business of what software development can do.
SDM is all about breaking down silos to focus on delivering the best, most impactful software possible and. But it’s not enough to deliver fast and without flaws. The core metrics you try to achieve have to be aligned and focused on delivering the quality software customers want and are willing to pay for. Creating a synchronized system based on common data, integrated tools, universal insights and streamlined processes helps organisations measure development effort to business value, resulting in making software delivery a core business process.