Costs & Benefits

How to optimise DevOps performance with a benchmark

by Anna Pschierer

DevOps is an attempt to combine the art of development with service efficiency. With a structured DevOps assessment, the collaboration of DevOps teams and their output can be evaluated internally and against the market to demonstrate performance and room for improvement.

Measure and optimise DevOps performance

If statistics are anything to go by, DevOps is the world's most widely adopted method of managing application services. Ideally, it may even be the best method for an organisation - but one-size-fits-all approaches usually have their drawbacks. The most efficient development and operations teams will always struggle if management complains about a lack of visibility and is uncertain about performance. A tailored DevOps assessment can show whether the move to DevOps is paying off - and where the friction points are.

DevOps Benchmark in Application Management

The aim of our structured DevOps benchmark is to evaluate application management organisations in terms of their efficiency, quality, speed and (process) governance. Whereas in a traditional operations or development benchmark, quantitative and qualitative points are analysed using various effort drivers as part of a market comparison, DevOps adds the crucial question: Does the concept actually fit my (IT) organisation? Or: Does our organisation fit with DevOps? In highly hierarchical companies, another model may be better, or there may be no significant need for constant innovation, rapid rollouts and continuous delivery.

Example of a DevOps assessment

For a large organisation in Europe, we analysed the performance of DevOps and compared it to market levels. All teams were using the Scaled Agile Framework (SAFe) and working on their own automated platform. It quickly became clear that the efficiency of the various developer teams was undoubtedly very high. One problem, however, was the connection between Dev and Ops, with operations taking a back seat to the priority of development. In addition, different IT Service Management (ITSM) tools were being used, which prevented an holistic view.
Data for the assessment was collected in joint workshops with the DevOps teams. In addition to traditional infrastructure and operations questions, the focus was on development and the interaction between the two. Relevant questions included quality of the code, criticality of the application and its age, which provide valuable information about the status of application management, especially in the overall context. Data validation workshops were then held with the teams to gather their valuable feedback.

DevOps comparisons and metrics

Internally available and measured KPIs showed DevOps as a common organisational principle does not guarantee the use of uniform KPIs. In addition to the DORA metrics, this DevOps environment measured flow velocity, item size and lead time to customer. Their KPI system had been set up centrally, but some values showed serious deviations. For example, in some teams, tasks were sitting in the backlog for a long time, because it was used as a 'repository of ideas'. However, this meant the centrally defined lead time could not be met.

Measuring or estimating DevOps overhead?

Time was also recorded differently: Sometimes the overhead duties in teams was 40 per cent, because it included run and change efforts. Other groups did not report overhead, or only estimated it roughly. In these cases, comparability is as impossible as targeted optimisation. Organisations should therefore pay attention to the extent to which their teams are working and measuring consistently. Otherwise, it will not be possible to make reliable decisions based on this data.

DevOps needs clear governance

Through our DevOps assessment, IT management has recognised a DevOps organisation of independent teams needs a consistent governance framework. The results are not intended to control or undermine self-organisation, but to make life easier for the teams: less administration, less duplication, standardised tools - and more time for innovation. In this way, the organisation can continue to focus on high efficiency in development which is the core of the idea: a transparent and controllable connection between development and operations. So working in DevOps is not only efficient, but also effective in terms of business goals and priorities.

Anna Pschierer

Anna Pschierer

Anna Pschierer has made her way from digitalisation of industry companies into IT consulting. Here, she drives the analysis and optimisation of appli­cation management with great commitment. She is passionate about the successful interaction between agile development teams and appli­cation operations.