Developer Productivity is Waste

The nerd fight about developer productivity

(0:04) Well, hello. I usually like to avoid having meta commentary if there's a recording, because that's a little weird - you've got to do that little "t equals some seconds" for the recording. But there have been some great talks and it's always nice to be here at DevOpsDays. I really appreciate it. So far, if the rest of the talks are good, even if mine's terrible you'll have a good time from the two that you've had.

(0:29) What I want to go over - and I assure you we didn't coordinate ahead of time, the previous talk and I - but it's great, because that gives you a kind of tools look at what things are definitional. It was really nice to look at. What I want to focus on: this started off as talking about metrics and things like that. I don't know if you read the same thing I do, but there was a great nerd fight last fall when McKinsey came out with this developer productivity metrics thing, which spawned all sorts of discussion. It wasn't as good as, I think it was maybe the 2012 bimodal IT discussion - maybe that was 2009 - which is fantastic, very choice. You should look that up if you haven't read it. I don't think anyone who was saying bimodal IT was bad paid the $5,000 for the Gartner PDF, but that shouldn't make you think it was not good. Good stuff.

(1:16) So this is a little bit of an evolution of what was going on way back then. But I think it's also thinking about what we are going to do in this era of DevOps. How can we prepare for it? I like the idea of DevOps X - that kind of has a nice feel to it.

The quick answers

(1:35) Now, the first thing when you're thinking about developer productivity and metrics that I think you want to know is: you just want to know which metrics to use. You're like, I don't know, do I really need to sit through 20 more minutes of this nonsense - can you just tell me which metrics I should be using? Of course, all the thought leaders and the masters are going to tell you you're a terrible person and you shouldn't be asking for metrics, which is what the rest of the talk will be like, so you can say I'm not a terrible person. But let me give you the answers right now. As I covered: it depends.

Now, whenever I encounter "it depends" I'm like, how much is this consulting bill going to be? I want to know - why don't you just give me the answers? But it is true, as I'll get into, it's very genuine that it's hard to apply lots of these metrics perfectly the first time. And especially since I imagine a lot of you are lower down the hierarchy, you've got to be really careful about just going in with some metrics that may have retributive sorts of actions based on not really analyzing that "it depends" much.

(2:41) Now, ideally - nonprofits and governments aside, but you can substitute in there what you would like - what you want with metrics is as direct a connection to money as possible. You want to say: am I being productive? Here's how much money I helped you make. Now, it's a little dangerous to say here's how much money I saved, because that usually means here's how many people I got rid of. But somehow you want to connect what you're doing to the actual outcomes, as we say in the business, the value, these sorts of things.

(3:07) But I think the problem with software, especially in what I would call - in a good way - normal companies, enterprises I'll say to bring that word up, is that the connection between the business and the IT shop, those people in the poorly lit, out-of-date sort of facilities, probably using machines that are six years old because the refresh cycle is out of date - you can kind of see that they're not really valued and thought about as contributing to the business so much. So that connection is very tenuous and difficult to make.

(3:41) And so instead we come up with a series of other metrics. This is kind of an overview - there's another one I'll get to later - but this is probably the last two, three years or so. If I could read that tiny type I could tell you.

(3:55) What we've really evolved from is this nice sort of "it depends" situation, where it depends, and here are the frameworks that will help you - no pun intended - figure out what your depends are. They ask you a series of questions, have you look at what you think productivity is and how it applies, and eventually you go down to one.

(4:18) And I'm going to very much simplify it: really, the state of the art in developer productivity is, are your developers happy? And if you can say yes to that, chances are they've got good productivity. I'm no academic, I just read PDFs, but that's sort of what a lot of this amounts to. Which is great, and it intuitively makes sense. The other way I think about it, back when I was a developer, is just stop interrupting me - that's a lot of developer productivity as well. But I would encourage you to look at these and think of them almost as sensing mechanisms for coming up with what the metrics will be, and the answer to what "it depends" is.

(5:00) The second thing, to give you the answers, that I'm going to build up to: what I've observed over however many years - it's good not to remember how many years - is that when organizations are talking about developer productivity, they're not so interested in, I don't know if you kids still use these terms, the inner loop. What developers are doing when their fingers are on the keyboard, or what apparently AI is going to replace at some point. If you're interested in another career that involves not coding, I could give you my tips on doing that - I had a similar thing in the 2000s. Anyway: not really the inner loop, but more what happens around the application cycle. Getting your code into production, going through all the governance, all those sorts of things. One of my theories is that if you focus on that a lot more, you probably won't really care so much about developer productivity, but you'll improve why you care about developer productivity - which is more about release cycles and getting features out.

(6:06) So before that, this is me. As you can tell, I usually wear that shirt, but my dog got very nervous and ripped a big hole in it, which is unfortunate, so my run on that shirt has run out. I've been really lucky for the past 10 years, and many years before that, to work at Pivotal, and then VMware, and now as we call it VMware Tanzu by Broadcom. I've had the chance to write lots of books and do lots of podcast stuff, I was an analyst, all sorts of nonsense like that. As a friend of many of us here likes to say: engage with my brand. I won't show you my various hair configurations, because it's been the same forever, which I stress out about. You ever notice when you were younger you were like, what's up with all the old people with the same hair all the time? And now I think I've become that.

Who's asking?

(6:54) So if you're worried about developer productivity and metrics, the first very important thing to do is to ask who's asking. This is not the most important thing - I don't like to rank things - but it's definitely top three, and it's really going to let you narrow down what you would do out of all the possible things.

In normal organizations and enterprises, the person who's asking is usually this guy - or gal, or non-identifying fantastic hair person. What you need to understand from them is that they're the ones who are responsible for the success of the business, or the organization. If you're in a nonprofit, it might be the success of satisfying our citizens' needs and renewing their passports with minimal impact, and not showing up in front of parliament or Congress to answer why things went wrong. But whatever those motivations are, the bosses, if you will, they're the ones who are first going to be looking at productivity and very interested in it.

(8:03) Now there are great reasons why they would be interested, and I'm going to shift into the commercial area. Hopefully competition isn't something that you have your governments doing too much - that tends to not go very well. But generally you might have competition in the industry you're in. We had bol.com up here earlier. Here's a joke: how do you know if an American's an expat? Because they tell you. So when I moved here to Amsterdam about six years ago, Amazon wasn't here at all, because bol was here and they were great. And they still are pretty great, they're very competitive. So from their perspective, the more they can get features out, the more their software drives things, the more they can succeed. Admittedly Amazon's pretty good now - order from Germany, which is great. The great thing about Europe is things ship around really quick. It's kind of like ordering from Kentucky when I was back in Texas, I guess. So they're interested in shipping features. They're interested in getting parity with their competitors, being better than their competitors, and what this really manifests as is just: ship, get features out. They're not really interested in updating your Java VMs and things like that. So that's the way they're going to be measuring things.

(9:17) And then of course, as I mentioned, they're interested in growth. They want to make more money, as many of us probably do - and if you don't need the extra money, I'll be available after this with a large bag, if you're satisfied with the amount you have. They'll also be interested in how we increase productivity in kind of a financial way: if we want to enter a new market or do more business, can I just get 10 more developers and I'm going to get better productivity? Or do I need to focus on making what I have productive? Do I need to do more with the same? This is the kind of thinking that went into the side of that nerd fight we didn't like, where managers of companies wanted to know how to program the system and get observability into how their software function was working.

(10:04) Now, the reason I would advise you to really think about your metrics is the other thing our white-toothed friend is thinking about, which is: I don't really understand the connection between all of this stuff. I've got one thing I can do, and that is pay less. So when I'm managing the IT function that I have - this is why in regular companies you see a lot of IT under the CFO, because the CFO's job is to be like, how about I don't pay that much. I'm grossly simplifying it, but from that perspective it's how do I get what I need at the lowest cost. Think about it: if it's August and you're in Texas, which is very hot, you don't want to paint the outside of your house on your own like my wife and I did a long time ago. You want to hire someone to do it. Do you hire the most expensive person because they're high quality and they've got artisanal brushes and they've figured out the perfect way to paint things and they've talked at conferences? No, you're going to hire the cheapest or the second cheapest person. It's the same kind of perspective that people will have.

(11:10) And again, this is what causes a lot of problems: from their perspective, the way to control costs is people. This is why they want to get down to individual developer productivity measurement. They want to know who are the underperformers I can get rid of, who are the overperformers I can retain and, if really cool, maybe give more money to. How do I manage that flow of people? And as we know, measuring developer productivity is very difficult. That's the main idea I want you to take away: if anyone asks you about measuring developer productivity, don't tell them it's only an inward-facing thing that you want to focus on - unless you're in the generative one, I forget the DORA three columns of success and doom depending on which way you go. If you're in a cool place, talk about it all the time.

(11:59) And then of course - I think it was called PSP, if we were talking about Delphi and Turbo Pascal from that era - there was this idea that you can track these things to improve your own craft and get better. It's good to keep up with those metrics, but that's more of an inward-facing thing that you have. Which is great, you should do that. Or not. But that's not really, as you can tell, what my concern is. I think the individual craftsperson can measure and do things on their own and it'll be pretty much cool.

What is "developer productivity"?

(12:31) So let's go over what developer productivity is, dive into it some more. Well, to emphasize the point: it of course depends. Which I think is a very accurate answer, but not a helpful one. It's not really going to push you forward into doing things.

(12:53) So thankfully, one of the great outcomes of this nerd fight was that Kent Beck wrote a two-part reply with Gergely Orosz. I should meet that guy, he lives here and our wives kind of know each other. Think about productivity in any area as basically the amount of effort I put in versus the outcome that I get - and can I lessen this as much as possible? How much of that productivity could be cost, time, friction. And the way you measure it is not just output, but really - if you'll forgive the marketing terms - it's the impact, it's the value to the customer, the money that you make, the time you reduce on that passport renewal, whatever it may be.

(13:33) So that's your basic framework for productivity. And then if you apply that to application development, it's all about this cycle - the product management cycle that the bol people went over in one of their slides. What's a problem or an idea, how do we implement it, how do we design it, how do we code it, how do we test it, then we've got to get it out to the users. And this part is the part that, when I was a programmer, we didn't do: we only monitored when people were suffering, with bug reports. But you actually monitor the people using it and see if behavior has changed, and then you say, is that good or bad behavior? This is the productivity cycle we have in software. You're putting things out there, you've got to see if you get the outcome that you wanted, and maybe cost. But the productivity you're getting through here is: are we actually making a difference?

The metrics buffet

(14:28) So in an "it depends" situation, what you end up doing is going to a buffet or a cafeteria. Or nowadays there are those hot pot places where you've got all these ingredients and you can pick from the metrics and throw them together and cook them yourselves, and feel a little weird the next day. Similar experience for metrics, probably.

(14:53) Let's go a bit more into what these metrics are, a history of them if you will. All of us here, we probably know the DORA metrics - there are four of them - and these are a good way of measuring that sort of impact and outcome. Depending on the values you have, these are some of the ideal states you'll be in. But they don't necessarily tell you about the productivity of what developers' activities are. They tell you if the machine, if the factory, is operating well. We're here at DevOpsDays, I'm sure you all have read the Accelerate book five times and got lots of bookmarks. They're great from that perspective.

(15:38) But more, what thinking about developer productivity and metrics has evolved into is looking at the activities - as much as that's disliked - of what developers are doing, but looking at it with respect to their day-to-day work, the flow that they have, the way they're thinking about things. These are just a sample from the SPACE framework, but it really starts getting to the point that here are all these attributes we can monitor, including well-being and the mental-ness, the kind of mid-era DevOps vibe that we had before we went into the observability era. All your sort of cultural stuff. It's a great framework for buffeting that stuff out.

(16:25) More recently, the - I'm going to call it the DevEx framework, I don't know if it has a title - from that same group came out. Again it's this buffet you can go to. What it's trying to get at, as I was saying earlier, is that we know when we look at it what a hygienic, well-running software factory looks like. What that looks like is this flow through it. And what it also looks like is: people are happy. Or satisfied, depending on how nuanced you want to be. So it almost starts with what makes people happy, what kind of environment you set up to encourage this flow, and therefore what metrics we're going to monitor that instrument that factory we have. And then that gets into all of the tools and the way you start fixing things.

(17:16) There's another group of metrics I first heard about at Monktoberfest last year, from Cat Hicks and friends. They're at Pluralsight, of all places. They picked up the great marketing advantage of the DevOps report, where they're like, what if we contextualize the market we're working in by doing some academic work in here, and think of the leads. They actually have a very similar sort of study across multiple papers of how you figure out that thriving that you want your developers to have, and all of the little attributes around it. So those are some good buffets for you to go look at. I think they're quite useful for sorting things out.

Tools: surveys, pipelines, platforms

(17:58) So, getting to the tools that you might use. And by tools I don't mean the excellent overview we saw of the configuration things, the IDEs - the paintbrushes, to use my old analogy. These are more the things in the outer loop.

The first thing - and I keep referencing it because it was great - if you think about what the bol people were talking about: product managing your factory, that loop. The thing about product managing is it's good to know your customer and what they're doing, otherwise you're just making stuff up, which is nice for poets and fiction writing but not really great for engineering sorts of things.

(18:45) So really, I think the state of the art of figuring out what your developers are up to - again, in a large organization with hundreds and hundreds of developers; if you've just got a handful you go have a 90-minute lunch with them and sort it out, which is probably fun - is just sending surveys out. This is a survey we used at Pivotal Labs, now Tanzu Labs, VMware by Broadcom or something. I worked with them a couple years ago to make it free. I think there's a CSV you can download, or you can copy and paste stuff. If you look over it, it's asking about tools usage and sentiment and about how long builds take, all of these attributes that are in our buffet.

(19:27) What you want to do with a survey like this - again, think about that loop of "we deploy a solution and then we need to actually monitor it" - is you send it out initially, and then depending on what period you want, maybe you send it out in a month, maybe a quarter. What you're doing there is you've got that instrumentation, you can see if you're improving things. You want the bad things to go down, otherwise you need to try again. Having this kind of input about how the productivity is going, based on attributes like this, gives you that good feedback loop, that sense of what to do. There are all sorts of other surveys, but that's the one I have on the slide.

(20:09) The other thing on that outer loop - and I always feel a little ridiculous bringing this up. I kind of count the start of DevOps as, I think it was 2008 or 2009, with the agile infrastructure presentation from Andrew Clay Shafer, as he fancies himself nowadays. Because of course we all have continuous integration and continuous delivery, or deployment, whatever, in place. But if you don't have that in place, then your productivity is going to be a little weird.

(20:46) Just as a refresher, this is what we're talking about - I'll probably say pipeline after this. More or less what we figured out, and this is why paying attention to this outer loop is important, is that in each of these boxes, especially the developer box, I think they kind of know what they're doing. They're really optimized. You could say they're locally optimized, or you could say they know what they're doing - in their domain things work really well. But of course, as we all know, it's those lines between that we're really all the problems are. The wait times, the dependencies, the passing authority over. This time of year you're in Europe, you're like, nothing's happening until September, so we might as well forget about the velocity of releasing things - to get past the architectural board you've got to schedule meetings, all that kind of nonsense, get your security stuff set up.

(21:41) So having these kinds of pipelines in place and automating them, as we've known and I'm sure as everyone here does, is extremely important for actually having productivity.

(21:52) And the history - I looked this up especially for this presentation. Grady Booch, if you remember him, wears a lot of Hawaiian shirts nowadays, which is fantastic. Maybe in 1994 he had his object orientation book. I think I read that book when I was like 17, and I feel like I was hazed by my fellow programmers for reading through that, and then I did object-oriented Perl, which is super weird in retrospect. So he wrote about this idea that we should do continuous integration because it'd be great. Then of course XP comes out in 1998 - I don't know if they got 15 or 20 or 10 sort of principles - and that also has continuous integration. Then we had the lovely ThoughtWorks people, as always going from radar to implementation at breakneck speed, they had CruiseControl. Then we had Hudson, then Jenkins. And then of course continuous delivery, represented there by a fantastic book. And then, as if to say you can't apply this everywhere, this guy Gary Gruver, who was working at HP at the time, was like: I do the firmware for printers and I can do CI/CD for that. It took him about a year or so, but he wrote that up there. So you can pretty much do this for any sort of thing, as I'm sure many of you are doing.

(23:09) But when you talk with your peers after this conference and they're sort of upset and they're like, our productivity is not so good - maybe you can be empathetic for them based on this kind of chart that I've been looking at. This is from the State of Agile report. Sadly they stopped asking this question, and it's got a little more - you know when you get a survey that's in landscape you're like, I don't know what's going on here. They've kind of removed some of the disclosure from it, but you can see I collected it up to 2021. Surveys, blah blah blah, they can be good, bad, whatever. But what I like about this one is that it's many, many years. What you see is the green tells you continuous delivery or deployment - someone smarter than me can tell you the difference between those two - and the blue is just let's say automating your build and your test. What's remarkable here, you could almost remove the numbers from it and say the worst thing is that it's flat. We're not necessarily improving as an industry. We're not somehow getting better. Which maybe is why we're still here having DevOpsDays. When we get this up to, I don't know, 75, 80%, maybe we can just shut it down and we'll have solved the problem.

(24:26) They did stop that in 2021, but thankfully - thanks to the largesse of the Linux Foundation - the CD Foundation picked up the slack here. Again, surveys: don't worry about the transition of the numbers, but you can see over the years there's still a small amount of people that are actually doing CI/CD, and it's not really improving over time.

(24:51) And just to emphasize this point - I like to do a little chart education at least once in my talks. Here's a tip: if you want to emphasize something, do a 100% chart instead of just a bar chart, and then just go crazy with the colors. Here what I've done is: never mind people doing CI, what's the percentage of people not doing it? Again, you are probably the 29% of people who do it - congratulations, that's good. But there are a lot of people who are just not automating their build and their test, which is ridiculous. So all this effort you're going to spend on developer productivity, you're going to have this great box of productivity, and then this. It's just going to languish and sit around somewhere.

(25:35) So that's why I start to think about the box that you have around things - the outer loop is important as well.

Stop building your own platforms

(25:45) Now, also, especially for a DevOps crowd - we probably know what a platform is, but this is a similar area where if you look at those lines between the boxes, you have a lot of issues. Thankfully there's this reference architecture from the Cloud Native Computing Foundation, the CNCF software factory, that defines what a platform is. Which is great, because then all of us vendors doing one-minute pitches, we don't have to give you a vendor pitch about what a platform is. We can be like: that. And it's wholesome and good and true. But all of us platform people, we pretty much do this. We just don't use Easter colors for things.

(26:22) But these platform people are building these out all the time, and that becomes quite the issue. You spend a lot of time on that, and it's not only the time you spend building it, but the variation you have across your hundreds of developer teams using things.

(26:51) And this was a thrilling moment that we had in the past five years, where we were like: let's strip down all this stuff, all the way down from the platforms as a service and the platforms that we had, and let's just build it all the way back up. This has really pulled in a lot of time building the stuff out. But I think we've emerged from building up from the Kubernetes layer, and now, if we want to focus on improving that kind of flow through the factory, we've got to really address this issue that's persisted for a while - which is that you weren't really meant to use this Kubernetes stuff, application developers, or maybe even DevOps people. It's kind of unclear over the years what the thought leaders of the Kubernetes community have thought, although there are some pretty clear statements here and there.

(27:28) But it's this idea of: if we keep building all these platforms instead of using more standard ones, whether they're open source based or whatever - and even worse, if we have 5, 10, 20 platforms in our organization - you're going to end up having all sorts of issues. Those lines between the boxes will elongate.

(27:44) And you see this in a survey we just came out with. We've done a state of Kubernetes survey for about five or six years, depending on how you count it, and you can see that the more platforms you have - we only ask four or more, but the more platforms you have, the more issues that you have. Anyone who works in infrastructure knows that the more different types, the more variations and variability you have, the more annoying it is. You've got to context switch between these different things, you have to understand them. And then you also add in - I can never remember the difference between a combination and a permutation, because I'm a liberal arts person - but you also bring in all the different development teams using all these different platforms and these different tools, and you're in a mess.

(28:28) So you want to go down to as few platforms as possible. And I think that comes back to the point about developer productivity. The tools are important, of course - you want to go to the buffet, figure out what metrics make sense for your organization so you can talk about the impact you've achieved. You want to have that inward-looking "how am I doing, what's my health assessment" for stuff. But then think about the bigger picture. Make the grand first-generation DevOps move, which is: ah, it's the end-to-end process. That's really what we're here to solve. Always pull back to that and ask, am I the 68% of people who don't automate their build, or the 29%? I don't think those numbers add up, but we can roll back the tape and look. And really think about how you can focus on that outer loop part.

Wrapping up

(29:14) Just as another endorsement, I think what the bol people went over today was great. What I really like there is they said not only product management, but they were like, design - which blows people's minds. They're like, how would I get a designer working with the infrastructure people and the operations people? But that's really what I've seen over the years: if you've got engineers, product managers and design people, you kind of engage in building up that nice outer loop.

(29:44) So then finally, just two things. These are the people - I'm sure there are others, but these are the people I read and listen to the most about developer productivity. They've got a great newsletter, and they also have a great series of YouTube videos. If you're in marketing like I am, I'm thinking of awarding them best thought leadership marketing campaign of the year, because they're really cornering this developer productivity conversation. I mean that in a good way and also a snarky way, but genuinely: if you follow what they're doing, you'll get a good sense about the evolution of developer productivity and metrics.

(30:23) And then of course, if you're interested in not doing ridiculous platforming, we have many fine pre-shaved yaks at VMware Tanzu by Broadcom, which I would endorse. You can check those out. And with that, thanks for having me. It was nice to see everyone and present here. I appreciate it, and enjoy the rest of the conference.