How to optimise DevOps performance with a benchmark

Measure and optimise DevOps performance

If statistics are anything to go by, DevOps is the world's most widely adopted method of managing application services. Ideally, it may even be the best method for an organisation - but one-size-fits-all approaches usually have their drawbacks. The most efficient development and operations teams will always struggle if management complains about a lack of visibility and is uncertain about performance. A tailored DevOps assessment can show whether the move to DevOps is paying off - and where the friction points are.

DevOps Benchmark in Application Management

The aim of our structured DevOps benchmark is to evaluate application management organisations in terms of their efficiency, quality, speed and (process) governance. Whereas in a traditional operations or development benchmark, quantitative and qualitative points are analysed using various effort drivers as part of a market comparison, DevOps adds the crucial question: Does the concept actually fit my (IT) organisation? Or: Does our organisation fit with DevOps? In highly hierarchical companies, another model may be better, or there may be no significant need for constant innovation, rapid rollouts and continuous delivery.

Example of a DevOps assessment

For a large organisation in Europe, we analysed the performance of DevOps and compared it to market levels. All teams were using the Scaled Agile Framework (SAFe) and working on their own automated platform. It quickly became clear that the efficiency of the various developer teams was undoubtedly very high. One problem, however, was the connection between Dev and Ops, with operations taking a back seat to the priority of development. In addition, different IT Service Management (ITSM) tools were being used, which prevented an holistic view.
Data for the assessment was collected in joint workshops with the DevOps teams. In addition to traditional infrastructure and operations questions, the focus was on development and the interaction between the two. Relevant questions included quality of the code, criticality of the application and its age, which provide valuable information about the status of application management, especially in the overall context. Data validation workshops were then held with the teams to gather their valuable feedback.

DevOps comparisons and metrics

Internally available and measured KPIs showed DevOps as a common organisational principle does not guarantee the use of uniform KPIs. In addition to the DORA metrics, this DevOps environment measured flow velocity, item size and lead time to customer. Their KPI system had been set up centrally, but some values showed serious deviations. For example, in some teams, tasks were sitting in the backlog for a long time, because it was used as a 'repository of ideas'. However, this meant the centrally defined lead time could not be met.

Measuring or estimating DevOps overhead?

Time was also recorded differently: Sometimes the overhead duties in teams was 40 per cent, because it included run and change efforts. Other groups did not report overhead, or only estimated it roughly. In these cases, comparability is as impossible as targeted optimisation. Organisations should therefore pay attention to the extent to which their teams are working and measuring consistently. Otherwise, it will not be possible to make reliable decisions based on this data.

DevOps needs clear governance

Through our DevOps assessment, IT management has recognised a DevOps organisation of independent teams needs a consistent governance framework. The results are not intended to control or undermine self-organisation, but to make life easier for the teams: less administration, less duplication, standardised tools - and more time for innovation. In this way, the organisation can continue to focus on high efficiency in development which is the core of the idea: a transparent and controllable connection between development and operations. So working in DevOps is not only efficient, but also effective in terms of business goals and priorities.