Can DevEx Metrics Drive Developer Productivity?
Developer experience, as it centers on human beings, is inherently sociotechnical. Yet, much of the work of “DevEx” and developer productivity focuses solely on the technical — despite the long-held truth that happy workers are more productive. Technical leadership typically concentrates on measuring the output of developers or the time it takes for them to complete tasks — which makes for a whole lot of story points, and not a lot of influenced change.
Last month, a research paper entitled “DevEx: What Actually Drives Productivity” went viral around the software consultancy world. It outlines an approach to understanding DevEx, as well as builds on a previously published actionable framework that combines developer feedback with that data from engineering systems.
Neither paper provides a secret formula but they aim to offer organizations potential areas to focus their measurements and improvements on. After all, developer experience and software delivery as a whole is dependent on factors at the individual, team and organizational levels.
Especially during a time of trying to do more with less, gaining insights into getting more out of the significant engineering cost center is a valuable endeavor. Here’s how.
What Is DevEx and How Can You Measure It?
“Developer productivity is more important than ever. I mean, everyone has been saying that forever, but companies right now are really focused on efficiency and doing more with the developers they have,” Abi Noda, CEO and co-founder of DX developer insights platform, told The New Stack.
At the same time, software development is evermore complex so that, “with all the different tools and technologies that developers use today, just doing your job is getting harder and harder,” he continued. “And then there’s also the shift to hybrid and remote work. People are trying to understand how does that affect developers and/or the productivity of their workforces.” This trifecta makes it the perfect time to dive into developer productivity and improving developer experience.
To the authors of this white paper, “Developer experience focuses on the lived experience of developers and the points of friction they encounter in their everyday work.” It’s not just about productivity, but increased efficiency, product quality and employee retention. DevEx has also been defined as encompassing how developers feel about, think about and value their work — not exactly easily measurable subjects, which may be why, unfortunately, most companies aren’t looking to measure them.
The authors coalesce around three dimensions of developer experience:
- Feedback loops – waiting time for developers to get done their work and how streamlined teams can shorten that time
- Cognitive load – in the ever-growing complexity of the cloud native world, organizations should look to limit hurdles to delivering value to customers
- Flow state – when developers “get in the zone,” with limited distractions — meetings, unplanned work, ad-hoc requests for help — they feel energized by a greater sense of purpose
These form the three angles of a triangle, all feeding into each other.
Early on, the paper cites a 2020 McKinsey study which revealed that companies with better work environments for their developers boasted dramatically increased developer velocity, which in turn correlated to four to five times the revenue of their competitors. It’s therefore presumed that the above three dimensions are highly influential to velocity.
What influences that developer experience comes down to 25 sociotechnical factors — including interruptions and friction from tools or processes — which are evaluated by survey responses. This data is then combined with existing data from tools, like issue trackers and CI/CD pipelines, as well as the traditional KPIs and OKRs. Another powerful DevEx metric, particularly during these leaner times, is Knowledge Discovery Efficiency or KEDE, which leverages repositories to identify developer knowledge gaps.
No matter which measurements work for your organization, it should be a sociotechnical blend of perceptual measurements — like how developers feel, as ascertained via semi-frequent surveys — and more concrete developer workflows.
Focus on the Person or the Team?
Developer experience is highly personal and contextually dependent, Noda said, which is why the framework is unique in focusing heavily on the individual. But that creates a challenge around how to measure the individual but work to improve the collective experience.
Indeed, the paper calls out surveys as an important “fast and accurate” measurement tool. After these carefully designed surveys — asking things like “Based on X, how likely are you to…” — are regularly run, break down results and Net Promoter Scores (NPS) by team and developer persona, advises the paper. Noda clarified in our interview that these surveys should be anonymized or aggregated. It remains unclear how anonymous it can be on the average “two-pizza team” of five to nine people, and if it really can be individually actionable to aggregate results.
A key measurement of developer experience is how good you perceive you are at your job — because feeling good at your job is highly motivational and signals both a lessened cognitive load and an optimized flow state. However this measurement brings its own slew of implicit biases that increase with intersections across demographic, role and experience.
After all, imposter syndrome is more likely if you are new to the tech industry or role and/or if you are from a marginalized group. Both of those circumstances would also make you feel less safe to reply honestly about flaws or hurdles. Add to all this, we are still in a time of tech layoffs, where morale may be down, but people may feel less safe to speak up. On the other hand, optimization, particularly for the individual’s flow state, would likely increase inclusion of neurodivergent developers.
All of these concerns should be considered within your DevEx survey design. The 2021 paper of the same authors “Actionable Framework for Understanding and Improving Developer Experience” is a more in-depth work following interviews with 21 developers and developer leads — though, it notes, despite efforts, it included only one woman. This paper cites psychological safety as the single most important factor affecting developer experience.
Psychological safety in this instance could be defined as feeling safe to speak frankly about your experience. “On teams with good psychological safety and culture, developers are more willing to voice and tackle problems in order to continuously improve developer experience,” the paper reads, while unsafer culture discourages developers from speaking up or trying to make proactive improvements.
Focus on Flow
Embracing your flow doesn’t just mean going fast — it’s as much about reducing friction and frustration for more focused and happy developers.
“Flow metrics is about finding a sustainable pace that allows you to keep going infinitely,” Sophia Ashley, scrum master at AND Digital, told The New Stack. Pointing to how flow metrics are often misunderstood, she said, “It’s not necessarily about speeding up. Yes, they can help you increase your velocity, but it’s about finding a pace that works for you,” making lead time also an individual metric. Once you’ve reached repeated consistency, she explained, you can then look to increase your pace, but at your own level of sustainability — endless growth is simply unsustainable.
In the capitalistic world of move fast and break things, she said that this more controlled growth can be a tough pill to swallow, but it falls in line with the change in an industry that’s embracing responsibility and environmental, social and governance or ESG goals. And it helps reduce the developer burnout that’s rampant in this industry.
Following the DevOps philosophy, for Ashley, flow metrics are teaching your team how to deliver sustainably. “A lot of companies want to do big bang releases,” and things break, she said. It’s more sustainable to do small releases to “teach teams to undo.”
Prior to joining the tech industry in 2018, Ashley was a physical therapist, from which she draws a lot of comparison, including post-injury training. “If they don’t exercise consistently, they will be stuck with their broken hip forever.” On tech teams, she continued, “Whatever we do, we stay flexible and we make changes that we can revert if needed, and that allows us ultimately to have this flow enabled that we don’t add damage to our company or environment.”
Progressive delivery is a series of technological solutions to help decrease cognitive load, allowing teams to roll back changes more easily. Observability and monitoring are also essential so bugs and causes of outages can be uncovered much faster.
Reflecting on the DevEx metrics triangle, Ashley said that it all comes back to that flow state. “Just being able to utilize your time well and keep working. That’s what developers want. Not being interrupted — context switching wastes a lot of time,” especially when developers are responsible for knowing several layers of the stack or are halted waiting for pull requests to be approved. “Work with users to understand the problems,” she said to shorten your feedback loops. And make sure you’re managing the developer cognitive load because “context switching like multitasking is not as efficient as doing one thing at a time.”
With her work as a consultant, she continuously runs some of the pulse surveys mentioned in the paper, asking:
- Are you happy within the team?
- Are you satisfied in your role?
- Do you think you provide value?
- Do you feel you are productive?
Is DevEx Just DevOps for the Individual?
It’s hard not to compare this DevEx approach to other widespread practices in the tech industry like DevOps and platform engineering. In part that’s because Nicole Forsgren is a prominent co-author of both these papers and of Accelerate, which is considered an essential DevOps text. But also this DevEx paper echos back to the three goals of DevOps:
- Shortening feedback loops with customers
- Systems thinking and flow
- Continuous experimentation and improvement
It’s just, while they both aim to increase the velocity of the software development lifecycle, DevOps focuses on the team while DevEx focuses on the individual. But, of course, optimizing for more developers to reach their flow states in turn should reduce the time to deliver value to customers. And by delivering value to customers faster, this in turn tightens feedback loops, reduces developer frustration and more regularly offers that dopamine boost of doing work that matters.
As established in Accelerate, DORA Metrics — deployment frequency, lead time for changes, mean time to recovery, and change failure rate — are as important than ever. DevEx just focuses on the individual’s contribution to these team, department or division metrics.
And then if you look at the next level up, the discipline of platform engineering observes and learns from the work of different teams to find behavioral patterns and especially blockers to the value flow chain. It aims to reduce, abstract and automate any demotivating, repetitive and non-differential work. It also further reduces context switching so developers stay focused on delivering value to the end users.
“Platform teams have to actually be understanding where the organization is at and what’s holding back productivity and make sure that they’re tackling those things and showing the impact of them by measuring and tying that back to the goals of the business,” Noda said. This is what distinguishes the platform teams that are adding value during economic downturn and the old-fashioned ones that just toss the platform over and are more likely to be cut right now.
Also, whether it’s borrowing developers, embedding within the app teams, or running lunch-and-learns and regular surveys, we know the biggest factor into the success of platform teams is reducing that feedback loop with internal developer customers, prioritizing them as your internal customers.
So as organizations look to increase developer productivity, at a time of likely reduced headcount, there could be a strong argument to examine the developer experience at three levels — individual, team and company-wide — to truly unlock the power of developer experience. And to run regular surveys that look to measure psychological safety, so the presence of problems is surfaced early and often at each tier.
