DORA Metrics
DORA metrics measure how teams deliver software: deployment frequency, lead time for changes, change failure rate and time to restore service.
What has changed is how we deliver. In the agentic way of working we now prefer, AI agents write a large share of our code. The classic four metrics assume each commit and deploy costs human effort. Once agents do much of the work, that assumption breaks:
- Deployment frequency goes up with trivial changes and stops being a signal of value delivered.
- Lead time for changes can shrink through revert churn without anything new reaching production.
- Change failure rate tends to rise, with many small, agent-generated failures that never show up as major incidents.
- Time to restore matters more than ever.
We are moving DORA metrics back to Trial to work out how to use them in our agentic way of working: which of the classic metrics still mean something, which extra metrics we need, and how to measure them without just rewarding throughput.
Beyond DORA, we want to evaluate Site Reliability Engineering metrics in a broader sense, such as the DevOps and SRE metrics described by Logz.io, to get a fuller picture of both delivery performance and the reliability of what we run in production.
The DORA Metrics (DevOps Research and Assessment) are a set of performance indicators that serve as an industry benchmark for evaluating the effectiveness of DevOps and software delivery performance.
They consist of four key metrics:
- Deployment Frequency: This metric measures how often a team deploys code to production. High-performing teams typically deploy more frequently, promoting smaller, manageable changes and quicker user feedback.
- Lead Time for Changes: This metric measures the time from when code is committed until it is deployed in production. A lower lead time indicates a more streamlined and efficient development and deployment process.
- Change Failure Rate: This measures the percentage of changes that fail in production. A lower change failure rate signifies higher code quality and reliable deployment practices.
- Time to Restore Service: This metric measures how quickly a team can recover from a failure in production. Quick restoration times demonstrate robust incident response processes.
Adopting the DORA metrics is advantageous for several reasons:
- Benchmarking: They provide an industry-standard measure of DevOps performance.
- Identifying Improvement Areas: These metrics can help pinpoint where our practices need refinement.
- Guiding Efficient Practices: The metrics can steer us towards practices that increase productivity and efficiency.
Several tools can assist in tracking these metrics:
- LinearB provides visibility into your development workflow, offering insights that help identify bottlenecks and improve efficiency. It can assist in measuring lead time and deployment frequency.
- Apache DevLake is an open-source tool aggregating data from various sources. It can centralize and streamline data collection, simplifying the calculation of DORA metrics.
- Swarmia offers analytics and insights into your software development processes. It provides detailed reports on various metrics, including the DORA metrics, helping identify trends and areas for improvement.
Creating a custom data lake can be beneficial for larger organizations with more complex requirements. A data lake allows for storing, integrating, and analyzing large amounts of raw data from various sources. This would offer us flexibility in calculating the DORA metrics and any other KPIs specific to our context, using tools of our choice for data extraction, storage, and analysis.
In conclusion, while the DORA metrics are a highly beneficial standard, we must remember to interpret them in the context of our teams, processes, and goals. They should serve as one component of a broader, holistic strategy for continuous improvement.