• Michael Cote, Staff Technologist at VMware Tanzu


• 30.11.2020 03:15 pm
Software

The race towards digital transformation has picked up pace during the pandemic. In fact, a new survey by KPMG and Harvey Nash indicates that businesses around the world have spent the equivalent of $15bn extra per week on technology as a result of investment in product development. But how do they measure success? As with everything in a company, product development is by no means immune from metrics. In fact, executives must rely on metrics, especially with hundreds of product teams, if not more.

However, as product development cycles run fast and experience constant iterations, it can be difficult to implement any numerical metrics. Qualitative metrics are incredibly helpful so long as company leaders have developed an understanding and intuition about the mechanics of using software to innovate. For example, one CEO at a large retailer began asking product teams what they’d learned in recent releases in addition to checking the status and budget of projects.

Here are three qualitative ways in which companies can look to define their success metrics:

Technical metrics

There are many technical metrics to use, most of them characterising how IT is performing. Even if you’re not in IT, understanding these metrics so that you can get a sense of IT’s overall health is important. Ideally, Line of Business (LoB) teams can delegate constant monitoring of these to the CIO and their team, but they should still take time to understand them. This provides the ability to address any problems and drive budgeting decisions based on performance.

For several years now, based on studies of high-performing organisations, the DevOps Reports recommend four key IT metrics. These are good to start with and evolve. I’ve listed them here alongside my recommendations for the ideal state, and where further investigation might be required:

• Deployment frequency: This is how often software is deployed to production for people to actually use. Target weekly and be concerned when it’s monthly or longer.
• Lead time for changes: After developers are done writing and verifying code, how long does it take to deploy to production? Target a week or less, be concerned when it’s two weeks or more.
• Time to restore service: When an error occurs in production, the length of time it takes to restore service. Target a day (an hour or less eventually), be concerned if it’s three days or more.
• Change failure rate: The percentage of changes that result in errors or poor application performance that require rolling an immediate fix. Target zero to 15%, be concerned if it’s more than 20%.

Of course, the ranges of good and bad for these metrics will depend on the type of business you're in the service expectations.

There are many, many other IT metrics you could follow. Ask product teams and operations staff what metrics they’d like to share, and consider what would be helpful for understanding overall IT health.

Business value metrics

The most useful metrics you can create and track are business value metrics: those that measure the success and improvement of business performance. By continually tracking, you can better appreciate how software contributes to revenue, not just how it burns cash. This also means you can begin determining IT budget by how it ultimately benefits the business.

Automation here is key. I’ve witnessed too many organisations relying on manual collection of business metrics, right down to needing to know that one employee has accidentally forgotten access to a core ERP system or magic spreadsheets. Too much time spent generating metrics will make them error-prone and out of date.

Whatever you choose, try to limit your metrics to four or less. For example, in transforming its apps, the UK Government Digital Services targeted four business value metrics: digital take-up, completion rate of forms and services requests, cost per transaction, and user satisfaction. These matched their goals of getting more people to use online government services, building services that worked first time, saving money, and meeting user needs.

Culture metrics

As you transform your culture, you’ll want to get a sense of the direction of progress. You won’t be able to measure it exactly, but you can model it to get a sense of how transformation is going. These metrics also allow you to detect if the organisation is backsliding or when problems emerge in the company’s ethos. Exact metrics used will depend on your organisation and the changes you’re making, but could include:

Employee NPS (eNPS): This metric is similar to the usual NPS, but instead asks: “How likely is it that you would recommend your organisation as a place to work to a friend?” Measuring this over time will give a sense for how things are improving or getting worse.

Staff belief in leaders and mission: This measures people’s alignment with the company’s culture and plans, but also the leaderships’ ability to communicate and work with staff. Of course, a negative rating could also show that the leadership and/or mission aren’t working.

Staff retention and churn rate: As your culture improves, driving up psychological safety, people should want to stay in their jobs longer. This should reduce, or level off, churn rates. However, you don’t want 100% retention. Getting new people and ideas into the system is helpful, equally so, allowing others who don’t (want to) fit in to leave.

The above metrics are a good starting place for most companies and can be built upon depending on feedback and success rates. Communication is central to this, along with alignment across teams. Being agile and working towards a collective goal will enable companies to achieve, and measure, true success.