Continuous Delivery

Podcast Transcript

Rebecca Parsons:

Welcome to the Thoughtworks podcast. My name is Rebecca Parsons. I'm one of your recurring hosts and I'm delighted to be here today with the authors of the book Continuous Delivery, Dave Farley and Jez Humble. Why don't each of you tell us a little bit about what you've been up to. So Jez, do you want to start?

Jez Humble:

Sure. So since leaving Thoughtworks, I've done various things. I worked in the US Federal Government in a team called 18F as part of the Obama tech surge and I went and co-founded a startup that helps research with Dr. Nicole Forsgren, and we did a whole bunch of research into how to build high performing organizations, we assessed companies. And we were acquired in December, 2018 by Google where I now work. I also teach at UC Berkeley.

Rebecca Parsons:

And Dave?

Dave Farley:

Yeah. Since I left Thoughtworks, I built one of the world's highest performance exchanges, worked in a trading company and helped them to adopt continuous delivery. And about five years ago, I started my own business working as an independent consultant advising organizations all around the world on improving their software engineering approach and in the context of continuous delivery. And I spend a lot of time doing that kind of thing and writing, blogging, and talking at conferences and stuff.

Rebecca Parsons:

Okay. So continuous delivery, it's one of those terms that seems to mean a lot of different things to a lot of different people. One of the things that was actually mentioned, by the way I should say, we are at the Delivery Conference here, which is a more technically focused conference organized around the principles of continuous delivery and DevOps. And I will let the these two rant on the relationship of those words, but that is one of the things that has been discussed is really how do you define continuous delivery in less than a few hundred page book? Do you want to take a stab?

Dave Farley:

Sure. So my favorite elevator pitch for continuous delivery is working so that your software is always in a releasable state.

Jez Humble:

Yeah. And I think when we looked at State of DevOps Report, we added to that, that you prioritize keeping the software working over building new features, is the other piece of [crosstalk 00:02:31].

Rebecca Parsons:

Okay. What do you think is the most important yet least well understood aspect of continuous delivery?

Dave Farley:

Nearly everything. So the current terminology is, the one that I hear most to talk about stuff in this space is CICD. And continuous delivery in the way in which Jez and I defined it in our book, incorporates continuous integration. Continuous integration is a part of continuous delivery. So that's one part that's misunderstood.

Dave Farley:

The other part that I think is important, if you genuinely want to work in a way so that your software is always in a releasable state then it's got to be tested, it's got to be validated, it's got to be deployable and the configuration needs to match. All of those things are part of being able to achieve. And the way in which I ... if you're starting from a blank sheet of paper, the first story that we build would be to flesh out a deployable tiny thing so that you have a stable surface that you can kind of start building on. You're kind of fleshing out the deployment architecture, the testability architecture. As well as the system architecture from day one of the development.

Dave Farley:

And I think that Jez and I are both biased of course, but I think continuous delivery is important because I think it's a disciplined approach to software development that when you practice it, if and when you take it seriously, and take the ideas of feedback seriously, it allows us to start really doing some quite deep things. We're applying scientific principles to software development. This is to my mind a genuine first-class engineering discipline for software development.

Dave Farley:

And on the back of that, that means that if you take this stuff seriously, you get higher order outcomes than if you just play at it. And the thing that is missed is that this is not a smorgasbord of techniques that you can kind of pick and choose from. If you really take this stuff seriously and take it all the way down, you get remarkable results, which Jez and Nicole's work has kind of shown. It's been demonstrated through measurable analysis.

Jez Humble:

Yeah, I mean originally for me at least the motivation was that the project I was working on a Thoughtworks, which really kicked it off for me was not having to go in at the weekends to do releases. That's what it was. I mean we took it from a process where you, the first release, we had to be in the whole weekend to a process where you can do push button deploys anytime of day and no one had to work outside the normal business hours to do a release, which is the question I ask audiences now to find out if they're really doing. How many hours do people have spent outside of business hours to do releases?

Jez Humble:

And unfortunately still lots of hands go up when I ask that question, which I take as a sign of personal failure, that we failed to actually get this done. Because it's the outcomes that are important. And I think the reason it's become popular is because there's that, there's also the fact it produces high quality software. It also reduces time to market and makes customers happier. It makes employees happier. These are all the kind of second order outcomes I think that we're talking about. I mean we've shown them to be true, but a lot of people think that it's about automating build and automating deployment.

Jez Humble:

And sure, that's a piece of it. But there's much more to it than that. And I think Dave, I agree with what Dave says, and there's a whole engineering discipline around comprehensive testing, being able to stand up environments. I mean a lot of people, as Dave think that, what is it? Get ops is the latest term of art. Well, I mean I'm all for people finding new ways to make ideas popular so that people do it because that's the important thing, that people do the thing.

Jez Humble:

But we were talking about this, Chapter 10 of Continuous Delivery, talks about infrastructure as code. This is all stuff we talked about and it's all necessary. You've got to do all of that stuff. It's about process improvement, architecture simplification. There's so much more to it than just the automation piece that people talk about.

Dave Farley:

Well, one of the things that delights me in this way of working is that those kind of second order effects, the kind of consequences of working this way. And I think that working, taking testing seriously, driving development from testing in a continuous delivery context improves the software architecture of the systems that we construct. The properties of software that make them testable are the same properties that we value as hallmarks of high quality in software, modularity, capsulation, information hiding, separation of concerns. All of those good things are what makes software testable.

Dave Farley:

And so it allows us to, as Jez says, demonstrably create higher quality software more quickly. The other thing that I think is wonderful that the work that DORA did and the State of DevOps Report highlighted, is that there's no trade off between moving quickly in small steps and quality. Those are not at opposite end of the spectrum. You create higher quality output by making change in small steps.

Jez Humble:

And this is the thing, it's not about trade-offs, it's about high quality and high throughput and high stability and higher availability. I know people might think that's snake oil, but basically that's exactly what happened to manufacturing with the Lean revolution. It wasn't that Toyota built crappy, cheap cars more quickly. They built higher quality cars, more cheaply and more quickly and really all that we've done is taken those same ideas, many of which have been popularized in software before us by people like Mary and Tom Poppendieck and Kent Beck and Martin Bell, and people like this, and just turn it into a body of work.

Dave Farley:

Yeah. I think one take on continuous delivery is that we kind of gave it a language to kind of organize some of those things, those great ideas from other people.

Rebecca Parsons:

And it's important to be able to get that kind of common language.

Dave Farley:

Yes.

Rebecca Parsons:

Which is then why it is always so frustrating when people misuse the term. But I do have another question for you then. You both emphasize the state of the software being ready to release as kind of the elevator pitch for continuous delivery, which says nothing about provisioning of hardware for example. If that's your definition, you could make the case that it doesn't matter if you've got a snowflake server out there if your software is all ready. So can you explore that a little bit?

Dave Farley:

Sure. One of the ways in which I talk about that and the infrastructure of code is that the environments in which your software operates is a dependency of your software. If you want to take this stuff seriously, then you want your software, a release candidate, to identify the platform on which it needs to operate.

Dave Farley:

So that's at the push of a button. You could deploy a version of the software that was running in production two years ago in a local test environment and reproduce that environment. To do that, you'd need to know which version of the operating system it ran on, what containers it was using, how it was configured and everything. You need to version control all of that stuff.

Dave Farley:

So I don't disambiguate those two ideas. I'm not talking about this in a narrow sense of just the application. As far as I am responsible for the ... if I'm writing software that runs on somebody else's environment, that's a slightly different game. But if I'm writing software as a service or something like that, then I am responsible for everything that my software depends on. And therefore I want to version control everything and be absolutely and precisely definitive about which software it is operating.

Jez Humble:

So in the book we talk about three foundations of continuous delivery. One of them is configuration management and configuration management is precisely about that, about being able to reproduce the entire stack from the ground up, from the operating system up.

Dave Farley:

Yeah. And this also is not a... so the book was published around about the same time that the early steps were taking place in the cloud I think, but the cloud was certainly not as widely used or as pervasive as it is today. And so this stuff is not just about the cloud. This works just as well in on prem data centers for organizations as in the cloud. But the cloud makes it more obvious sometimes that you should be automating all of this stuff.

Jez Humble:

Yeah. It's much easier to provision an environment through a tab API then it is on like BACS, but you can still do it on a BACS if you want-

Dave Farley:

Absolutely.

Jez Humble:

... and there was talk yesterday, Delivery Conf, about people doing continuous delivery on backs, we had people do it on mainframe system, we could have people do it on firmware. All the same principles apply. And the firmware one was really interesting because they built an emulator so they could write automated tests, and they'll run against this printer firmware. And again, if that's configuration management.

Dave Farley:

I worked with a team that did FPGA programming for trading systems and we did continuous delivery for FPDA program, which is even on a lower level than firmware. It works in all ... and I think it worked for some important and profound reasons.

Dave Farley:

Jez is a proper physicist. I'm a Popular Science reader. I'm a pretend physicist, but I genuinely believe that what we're talking about here is the application of the scientific method to solving problems in software. And that's important because the scientific method is humanity's best problem solving technique. And I would argue that the distinction between kind of agrarian civilizations and the high tech civilization that we live in today is fundamentally as a result of engineering.

Dave Farley:

That the application of that kind of reasoning to solving practical problems in the world, that was the exploded map two or 300 years ago. The fruits of which we saw explosively take off through the 20th Century and the 21st Century. That's where our high tech civilization comes from, applying that kind of reasoning. So it allows us to solve problems with more discipline and with more rigorous and it allows us to solve therefore more complex problems.

Rebecca Parsons:

So let's shift gears a little bit. There's been a lot of emphasis these days around, it sometimes it goes by the name testing and production or doing more things, watching the live system, as opposed to sort of the testing run-up. How do you feel that approach fits in with continuous delivery?

Jez Humble:

I think it fits in great honestly, and this is one of the areas I think there's been a lot of evolution since the book came out. Is how do you operate reliable distributed systems that are secure and so we've seen things like distributed tracing and observability, and whole lot of tools come out to manage that. And I think that fits very neatly in with what we talk about in Continuous Delivery.

Jez Humble:

I think from a technical level, basically it comes from what happens when you build ... fundamentally you're dealing with components, and the question is do you combine those components at build time into large packages that you deploy or do you independently deploy those components and have the binding happen at runtime? And when the binding happens at runtime you've got this situation where actually a lot of the behavior is emergent and you have to have a lot more instrumentation in production to be able to understand what's going on.

Jez Humble:

So if you've got a microservices architecture, you've got to have a distributed tracing otherwise you're never going to be able to keep up performance builds. If you're deploying big packages where all the combining and module happens at build time, you don't have to worry quite so much about that, but you certainly have to focus a lot on your pre-deploy testing. So I think there is a certain trade off in terms of how you deploy, whether it's binding at runtime or build time.

Jez Humble:

I think the things to be careful of is a lot of people have got this kind of misconception that runtime testing means you don't have to do pre-deploy testing. And I think that's totally false, you still have to do all the things that you did before. You still have to have unit tests and acceptance tests and performance tests and all these other things. That doesn't go away. But there is this extra set of stuff that you're doing.

Dave Farley:

I think there are different spheres of learning. There are different things that you want to learn. You learn from those in different places. So I completely, 100% agree with what Jez says in terms of any learning that you're ... testing in production is important. The empirical discovery of things that you haven't thought of is a fundamental learning opportunity that companies dismiss at their peril.

Dave Farley:

But you also need to be disciplined in approach and be able to do the kind of, I keep using the term engineering, to make sure that your software is doing what you think it is before you release it to production. And there are well established techniques that do that. There was a nice survey from a USENIX Conference a few years ago that looked into production failures. And I'm using sidebar, what's the most common line of code that a production system fails with? It's a comment saying ‘should insert exception handling here’.

Dave Farley:

But they came to the conclusion in this academic research, it's a kind of meta-analysis of production failures, but 70% of the failures would be resolved with just simple testing. If you adopt a test driven development, that eliminates 70% of production defects. Think for a moment the efforts associated with that 70%, the triage, the tooling around, managing huge lists of books and all of that kind of stuff. And I think that accounts for the kind of 44% of our extra time that the elite teams spend on doing new work, that the Accelerate book and the State of DevOps report identifies, the people that practice these sorts of things.

Dave Farley:

So applying techniques like test driven development and behavior driven development and those sorts of techniques to thoroughly test stuff is important. My careers post Thoughtworks and while I was at Thoughtworks, in some respects was largely in organizations that where system failures were probably more important. So I don't do some work with company's supplying medical systems and systems, hardware and software systems that can kill people, and there's a level of diligence that should appropriately be applied in those circumstances. You need to protect people and make sure that the software is first of all safe and it's not going to harm anybody.

Dave Farley:

And so I think you can probably, and there are some circumstances where you could probably get away with being a bit more lax because you could accept some software failures in production, but even in those organizations, if you really want to move forward quickly, you do that by doing the best quality work that you can. And you do that by testing enhanced as well as learning the lessons in production. And then the lessons in production are a different set of problems. They allow you to understand how your product to really land with your users, what really works and all of those kinds of things.

Jez Humble:

I think over the last, I mean as we've seen, comprehensively that you can use this stuff effectively in all kinds of domains, especially regulated domains and I've worked in government and Dave's worked in financial services. I think actually the thing people miss is that this stuff is if anything more important in those domains. You actually need this rigor more in those domains.

Jez Humble:

So for example, we talk about the four metrics that the DORA research program discovered. Lead time, you want to shorten lead time, you want faster through frequency, you want short time to restore service, and low change fail rates. And high performance do better on all of those axis. But think about just lead times. If you're building a system where if the system goes down, that's actually a big problem. Short lead times are really important. You've got to be able to fix those problems in a short and predictable time. When a new vulnerability comes out, you've got to be able to patch all the effected systems and redeploy them in a short and predictable amount of time.

Jez Humble:

What gives you the ability to do that safely? Continuous delivery. The same things that make it fast, make it safe. One of the things that I often talk about is are you using a different process from your regular process to do emergency fixes? So, okay, you're using a different process. What are you skipping? Well, typically what you're skipping is comprehensive testing because it's takes too long. So how good an idea is it when you've got a problem in production to skip a bunch of your testing? Probably not a great idea. So one of the things is you should be able to use your regular process for making emergency fixes because your regular process is so efficient and effective.

Dave Farley:

Yes.

Jez Humble:

And auditors love that stuff.

Dave Farley:

Yes. Yes. My experience in working in regulated industries is that there's always a degree of skepticism when you start talking to the regulators and then as you start describing the, I keep using the word discipline and rigor and that kind of sends the wrong message, I don't mean that this is there are people whipping people to do this stuff. But the thoughtful application of some disciplined approach to solving problems. When you start describing that, very quickly you start to win those people over in my experience.

Dave Farley:

And I wrote a blog post about this a little while ago. One of my clients started talking about continuous compliance and the conclusion of my blog post was that I've worked in regulated industries at Thoughtworks and before for a long time. And if I'm honest, my observation on the whole is that in the absence of a continuous delivery, people are always only paying lip service to the regulation. All of the organizations that I have ever seen have a stack of defects that the regulators are on their back about and that they never get to fix. In regulatory failings.

Dave Farley:

Continuous delivery flips that. I was involved in this project where we built one of the fastest, highest performance exchanges in the world. It was harder for us to release a change into production that was not compliant than it was for us to release the one that was because a deployment pipeline enforced compliance. We would have had to have manually broken the deployment pipeline to release a change. And I think this is important. If we care about these mission critical, safety critical systems, I think this kind of engineering discipline is what you must do to be really doing a diligent job in those sorts of industries.

Jez Humble:

And I think that what's happened is ... I asked when we were doing our keynote yesterday how many people in the audience are doing CDD? And it was about 20% of the audience. And I ask this question a lot when I do talks and it's consistently, wherever you go, whether it's Silicon Valley or China or anywhere in the world, big companies, small companies, it comes back around 20%. I mean, it's kind of the last agile practice. And everyone's very quick to talk about CI/CD in the context of build and deployment automation. But not many people are doing CDD and that's really important. And there are the people who say they're doing CI, most people aren't actually doing CI as well. So I hate to be a kind of curmudgeonly dude-

Dave Farley:

I thought that was my job.

Jez Humble:

We've got to stop finagling each other. But it's the usual thing. People do the things that are easy and then don't do the things that are hard because they're hard.

Dave Farley:

Yeah. That's what every single one of my clients that I've ever worked for wants is to carry on doing everything the same as they did, but get different results. And that's what Einstein defines as insanity. You can't do that.

Rebecca Parsons:

Yeah. That doesn't usually turn out to work very well. So where do you see continuous delivery going? I mean, you've talked a little bit about the different kinds of platforms, the mainframes, the firmware, the FPGA's. Tell me a little bit about what's the next thing, if you will, in continuous delivery or is there a next thing?

Jez Humble:

So I'm not an early adopter, so I'm naturally suspicious of new things. And most of what I think it's going to happen in the future is what has happened historically in the software industry, which is that new stuff keeps getting invented. And when that new stuff gets invented, we forget all the lessons that we learned and we have to relearn them all from scratch.

Jez Humble:

Which has consistently happened. I remember when mobile phones came out and there was no way of doing automated testing, we had to rebuild that from the ground up. And every time new technologies come out, again we have to rebuild from the ground up new ways of doing testing for them, which is great for Thoughtworkers because another way to build a new testing point, like, "Yay! Let's build a new CI tool in a new language. Yay."

Jez Humble:

But I think people think about technology as kind of this forward progress. And actually it's not like that. And I think I still have a copy of the Human Side of Enterprise from McGregor, which was written in 1960 talking about Theory X and Theory Y. We still don't have that sorted down. So I'm not at all convinced that there's a shiny, bright new world where we're done with this now, what's the next thing? I mean ...

Dave Farley:

I double up on your grumpy old man-isms. You're less of an old man, but I-

Jez Humble:

I'm working less.

Dave Farley:

... I think that's the case. One of my observations is that we are terrible at even eliminating bad ideas, let alone making progress. I'm currently working on a new book on the topic of software engineering. I don't know what the title is yet, but I'm working on some of these ideas and I did some research into the history of software engineering and some of the stuff that we talk about in the context of agile thinking, extreme programming, continuous delivery was written about in 1968. There's a description of test room in development from Alan Purvis at the NATO Software Conference in 1968 that's absolutely spot on if describing test driven development as far as I can see it.

Dave Farley:

And so I don't think we're good at the learning. The thing that I am slightly positive about is that I think that the first, this sounds a bit hubris, but I think continuous delivery is an important idea and not because we had it, but I think because of its impact on our ability to do cleverer things with software. And I think our book was a successful as it was and the idea has been as successful as it was because people have noticed that it worked.

Dave Farley:

And that's not nothing. That's a good thing. So even those people that are kind of paying lip service, I remember when Rebecca and I in the other day when we were working together at Thoughtworks, we'd be going into organizations and we were saying, "You probably don't want to look what we're doing because it will scare you, but we can fix your problem." It was so out there, the stuff that we were doing, the extreme programming and stuff was seen as scary.

Dave Farley:

Now everybody aspires to that kind of stuff. They pay lip service to it, but it's not really there, but now everybody would profess to be having a continuous delivery strategy or whatever. And that's a good thing.

Rebecca Parsons:

It's progress.

Dave Farley:

It is progress and so that's good. My belief because I think that the reasons why it works are quite deep and the kind of theme in my book that I'm writing at the moment is that software development is primarily about two things. It's about learning and it's about managing complexity. And we need to optimize to be good at those two things. And continuous delivery in particular is wonderful at allowing you to do both of those things.

Dave Farley:

It sort of table stakes that allows you to build excellent learning and excellent management of complexity on top of. And it goes back to this other thing, this application kind of scientific rational thinking and applying that to solving problems in software. If I were correct in those things, and I think that is durable whoever's talking about it or thinking about it. So it's not just because you and I wrote a book about it, but I think it's important because this is how humanity makes progress in different fields. We make progress by, we're finding out how to be better at them and we do that empirically over time and kind of synthesize the mechanisms and practices around certain behaviors to allow us to adopt these more empirical approaches to learning and discovery. And I think that's what we hit on I think by accident.

Jez Humble:

We set out to write like a quite niche boring book about build and test automation.

Dave Farley:

And we learn these lessons by getting it wrong lots of times, which is one of the beauty of science is that you trust the effect, when it gets wrong, that's where you really learn. But I think for that reason it's important. I said this a few times, half jokingly, but half true, I think that in 50 years time, even if it's the robots that are writing all the software, it will still be iterative, it will still be incremental, it will still be based on experiments because that's how anybody, anything solves genuinely hard problems. Evolution approach to learning is a wonderfully powerful information tool.

Jez Humble:

So I think it's mainly about well, what are people going to do?

Dave Farley:

Yeah.

Jez Humble:

And who gets to use it?

Rebecca Parsons:

Okay. Well thank you again for joining us. It's been a pleasure catching up with both of you and I look forward to reading the book when it comes out.

Dave Farley:

Thank you very much.

Jez Humble:

Thanks for having us. Thanks.

Dave Farley:

Thank you.

Mike Mason:

Join us on the next episode of the podcast where I'll be talking to Pat Sarnacke about his 22 year career path at Thoughtworks.