Jim Highsmith: a 54-year agile journey

Neal Ford: Hello, and welcome to the Thoughtworks Technology podcast. I'm one of your regular host, Neal Ford, and I'm joined today by Rebecca, another of our host.

Rebecca Parsons: Hello, everybody. My name is Rebecca Parsons, Chief Technology Officer for Thoughtworks. We are joined today by our guest and now retired Thoughtworker, Jim Highsmith.

Neal: You may know him as one of the signatories of the Agile Manifesto. A well-known figure in the Agile world, and that is exactly our topic of conversation today. We realized that as Jim was retiring, we wanted to grab him before he wandered away too far from the software development world and got too comfortable, and talk to him about some of the observations he has made.

A lot of people assume agility now because it's been around for a while, but boy, that was not the case when things got started. Jim has a unique perspective on how this got started and where it is now, so we wanted to talk to him about that. Welcome, Jim. We're glad to have you today.

Jim Highsmith: Thanks, Neal. It's great to be here again.

Neal: Let's talk about some of the things that you seen in the software development world. When did you start doing software development?

Jim: In about 1966, when I first got out of college, out of engineering school. One of the things that I've been thinking about a little bit over the last couple of months as we've reached the 20th anniversary of Agile is looking back over a little bit of history as to how we got here and looking for some themes that might be important. I really settled on two themes through the years. One of which is, is it the chicken or the egg? What I mean by that is, what came first?

Agile software development or the need for business agility? Or did they thrive each other? The other thing, too, is the fact that the real thing that agility addresses is uncertainty and how do you go forward in an uncertain world and how has that uncertainty increase over the last, say, 20 years. I really went back and I looked at this as my 54-year Agile journey.

Rebecca: That's quite a journey.

Neal: Which came first, the chicken or the egg?

Jim: Well, my answer to that is I think they really co-evolve. If you think about it, or looking back at the mid '60s, it was all about mainframe computers, actually, very small mainframe computers, punch cards, reports. There was very limited interactivity with the computer. Programs were mostly in COBOL and Fortran. The software engineering term was born during the late '60s. If you look at business itself, it was slow change.

In that period of time, you had a job for life if you went to work for a big company like General Motors or other big companies. You tended to stay there for an entire career. They began automating business processes, which is internal types of things. The story I remember from that era that epitomizes for me, the conservative nature of business during that period of time is-- Before I graduated, I was in Atlanta. My father was a civil engineer.

He invited me to come down to his company and talk to some people about what it meant to be an engineer. I said, "Great, I'll come down." I dressed up in my suit and my tie and I went down to talk to my dad and these other people. Afterwards, my dad was upset with me because I had worn a light yellow shirt rather than a white shirt. That epitomizes the conservatism of that era. In the next decade, my dad turned into an old hippie, so he's changed with the times too.

Rebecca: As you were talking, automating the business processes, these were well established processes as well. Business changed slowly, which meant that the requirements on business processes change slowly as well. I think there was far less uncertainty in what you were doing when you think about things like an accounts receivable system or an accounts payable system. There are pretty well understood rules for how to do those things. I think we all started with the low hanging fruit, if you will.

Jim: Yes, that's right. During the '70s, mainframe computer capability increase, but it was still a pretty primitive interactions. We were automating business processes themselves. Like you say, there was lower uncertainty because it was like you go interview the people in accounts receivable and see what they do on a day-to-day basis. You can write that down on a set of specifications. During the '70s, we started thinking more about software engineering.

During that period of time, the software engineering techniques were structured development, much structured requirements definition, which is a form of organization and diagrams. I remember one of the people that I worked for, Ken Orr, a number of years ago. We used to do things like bubble diagrams for a structured requirements definition. He went into an organization to do some consulting, and they had bubble diagrams all over the walls.

They were real proud of their bubble diagrams, and they asked him, "So what do you think?" Ken says, "I think you're lost," because they got so tied up in doing diagrams, and that was the downside of some of that. It emerged into the '80s when we began to get terminals so that we could actually interact with our computers through terminals. These were what they call green screen, text oriented.

It was a minimum about interaction, but those helped with building larger systems, but still mostly internal processes that we were automating during that period of time. We began what we called a waterfall methodologies. That actually came from a misinterpretation of a paper that was written about waterfall, but the case may be-- Again, the epitome during the '80s of what happened was called information engineering.

It started at the top of the company with big plans. Oftentimes, people would spend 18 to 24 months doing planning for information engineering, much less actually getting anything developed. Towards the end of the '80s, that became very much too slow, we're not getting anything out. We began a little bit, through the end of the '80s and into the early '90s, to change because of the uncertainty and speed that business was moving at during that period of time.

Neal: Now, I know markets has always overwhelm expectations, but what was the expected life span for a piece of software you'd spend 18 or 24 months designing? How long did they expect that to last?

Jim: Forever. [laughter] That was pretty much it. It's really interesting to go from the '80s to the '90s. In the '80s, people thought that software was going to go on forever. In the '90s, which we'll get to in a few minutes, rapid application development started because of the need to do things faster.

What happened with RAD is that people didn't really look at quality much anymore because the thought was the software is going to last for six months and then we'll go onto something else because things are changing so fast. You went from one extreme to the other between the '80s and the '90s.

Neal: Before you leave the era of the very formal design stuff, before the kinds of things we had to build in the market pressure really swamped everything. Was it ever really effective? Did you ever work on a project where you thought this is a really well- everything is working really well here, and things seemed to be going great, or was it never all that great a fit for building business systems?

Jim: Well, my experience with methodologies over the years, all the way up through Agile, is that each and every one of them worked to a certain extent and advanced the state of the art. Each one of them failed. Agile has failed in some instances just like all these other ones failed. Interesting during that period of time, most change management was done from the top down.

I remember working for a telecom company or with a telecom company during the mid '80s, if you will. I helped them install their fourth waterfall methodology in 10 years. They thought the next one would work, whereas the last one didn't work. The other thing that happened during that period of time as we were automating internal processes is they were very siloed. You did accounting system for accounting. You did manufacturing systems for manufacturing. You did accounts receivable system for accounts receivable. The interaction between applications was minimal during the '80s which gave rise in the '90s to things like SAP, which were integrated systems that you purchased and installed rather than wrote yourself.

Rebecca: Another thing that I remember from the '80s is you had a very clear split between people who did business processing, business IT, and people who did scientific computing. It wasn't just the COBOL-Fortran language split. There were different platforms. Even in the '60s and '70s, digital equipment corporation was heavily used in engineering, but it was almost seen as taboo in the business world.

No, we need our IBM. You also had Sun Microsystems. You had SGI, and you had all of these different systems that were more thought of from the scientific computing perspective than from the business perspective. I spent a lot of my time in the '80s and early '90s in more of the scientific computing realm. We didn't talk about things like waterfall or anything like that. Methodologies were what other people worried about.

We were focused much more on things like performance. I remember one debate about using this brand new language C++ and how good it would be for scientific computing because it's only a 50% performance penalty from hand tune Fortran and assembler. It's like if it already takes 24 hours to run, I can't afford to pay you 50% performance penalty. I don't care how good the language is. There was a real separation of what kind of things you worried about in those two different worlds.

Jim: Yes, it really was. During the early '70s, I worked for Exxon in a refinery, and we had two different computing groups in that organization. We had business computing group and we had a scientific computing group. They would do all the stuff related to the science of refining. One of the interesting things that happened was one of the people from that group, from the scientific group, came into the business group initially as a developer, but then as a manager.

He was used to Fortran. In Fortran, you had very short data element they use. They just didn't have much space. He wrote a payroll program using Fortran notation. He wrote it in COBOL, but he use very, very, very short variable names. It was a mess. Maintaining that system was really awful. Another thing that happened during that period of time which was that security was very, very minimal.

I remember when I first went to work in an organization, there was a guy named Ed, and Ed had his room full of card decks. One of the things that Ed did was he was the only person who knew how the end of year accounting systems ran and he had card decks for each of the things that had to be done in his desk. If something happened to Ed, we would not have been able to close the books at the end of the year. This is the wild days back in the early '70s in particular.

Neal: I remember one of the asides-- I've never worked in that era, but I've talked to someone who did, and he said that the worst thing that could happen to you in that era is be carrying a stack of cards and trip and fall and have to reorder them. The nice side effect of working in that era was that the computer rooms were always very air conditioned, even in the summer because the machinery ran so hot, they had to keep it cool. You're always guaranteed good air conditioning, no matter where you were.

Rebecca: Well, it also really affected how we think about programming. When you had to write your programs out on those wonderful key punch sheets and then turn them into the key punch operator and get your cards back and then go and present your deck to the guardian of the computer who had to feed it in and give you your results. You were really careful about thinking, did I actually get this right or not? We did all kinds of desk checks, before ever trying to run anything. The feedback loop was so long that you just thought about programming different. You wouldn't just put something together to experiment and see how that worked because it was going to take you four hours. What are you going to do for the four hours waiting for the results to come back?

Jim: Yes, it was really card decks in and printed reports out. That was the interface with computers during that period of time. There was no online terminals. It was just paper in and paper out, and like you said, it was very slow. I even worked on a system prior to that with the Apollo program where the input was not cards, it was paper tech. That was even worse to work with than cards. At least when you made a mistake with cards, you can change one card. It was pretty primitive back in those days.

Neal: What methodology stuff were you doing in the '80s? We've made it in our chronological journey up into the '80s now. What did software development look like in corporations in the '80s?

Jim: Well, what happened in the '80s, and I actually did a number of what you might call waterfall methodologies during the '80s. I actually built some systems using the waterfall. I won't say the projects went perfect, but they did turn out some results. In the '80s, we started to get better many computers. It was beginning of the PC era, in the early '80s.

Relational databases, you began to get terminals that would actually allow you to interact with the system a lot quicker than, of course, cards and paper. We began getting case tools, computer-aided software engineering tools. The problem with those is that they basically mirrored the waterfall lifecycle at the time with very little feedback loops. It was like you had to get everything right the first time, and then any subsequent step.

They really flourished, but it was a very short flourish. Still, the corporate world was changing, and there was uncertainty, but it still wasn't to the level of it that it was later. For example, in the beginning of the '80s, corporations had huge planning departments focused on business planning, five-year plans at great levels of detail, and so they had big planning staffs.

We were still doing automating internal processes and then trying to connect things together. I can remember one company I worked with in that period of time, they had a glitch in the payroll system. Somehow, that fed back into an inventory control system and nearly shut down a manufacturing line because of this interaction that was improper.

We were just beginning to hook these systems to each other and have really bigger, more complex systems. A lot of it became too much for a lot of companies to do on their own, and that's why you got the rise of integrated financial and manufacturing systems during the late '80s and early '90s.

Neal: Yes, I remember that transition from mainframes to mini and smaller computers. I had a college professor, I was in university in the late '80s. He told me if you really want a future in computing, in business, you need to focus on mainframe computers because those mini and microcomputers are fun, but there's no real future in that. Fortunately, I ignored his advice pretty thoroughly. His advice expired pretty quickly after that [laughs].

Jim: Yes, I'm sure that people today and a lot of the programmers today, developers today have never done anything but Agile. It's hard to think back. They've always had these systems that have high degrees of interactivity. It's hard to think back to the times when things were pretty cut and dried and not so interactive.

It's interesting, if we get into the '90s, we began to have client-server systems. One of the things that happened was people getting a big rush to do client-server systems, which were more interactive kinds of things. Almost immediately, the internet took over and killed client-server. That was a double transition for a lot of people that took some time and energy to do, as well as getting ready for the turn of the decade to the 2000s.

Rebecca: One of the other things I remember from that time as an industry, we seem to have a pendulum that goes from highly centralized and then highly decentralized. You had all of these tools as more PCs became available. You have shadow IT with critical business processes being run on an access database, sitting under somebody's desk and all of that. That was a big part of it during that time because IT had been used to working slowly, and things were changing more rapidly.

Now, you have this proliferation of tools that were relatively accessible for people. They would be building these applications and running critical parts of the business. I know in the 2000s, a big part of Thoughtworks's business was going in and replacing these little access databases with real pieces of software because people had recognized just how critical it was, and it wasn't going to be able to scale to the needs of the business.

Jim: Right. I knew a guy in the '90s who had worked on a mortgage processing system for banks. He did- I can't remember who it was, it was on a spreadsheet- but it was one of these user oriented tools that he did this with. They looked at this a couple of years later and they found out that he had made some mistakes and that they were losing business on every mortgage they took over.

That was the time when there was a lot of that thing going on. It was really interesting. There were two kinds of things going on during the '90s. One was something called business process re-engineering, which was really big during the '90s, particularly in the early part of the '90s. Again, it was a big systems, a deeper look into automated processes.

Then on the other hand, there was the rapid application development school which really was do it fast, do it quick, don't worry about quality right now because we're going to throw the software away. There were these two conflicting but different kinds of things going on during the '90s. Then you had beginning of the internet era.

Amazon was started in July of 1994. Netflix in 1997, so you're beginning to see the connectivity of the internet began to intrude. What was going on was a move from internal systems to external systems that actually your customers would deal with and your clients would deal with and your users would deal with. It set up a whole different dynamic because those kinds of things, you didn't have good requirements for it.

It wasn't like analyzing internal process and automating it. It was automating stuff that we'd never done before. That's why you had a greater degree of uncertainty. This is why the rapid application development took hold. Then, again, the forerunners of really the Agile movement started during that time. Adaptive development Scrum XP, all started around middle of the '90s and began becoming more and more important.

Neal: Yes, that's when you started hearing the phrase "Extranet". I can remember PC Magazine would have the year of the extranet as a big topic exactly to that point of having to start exposing some of your soft underbelly to the scary world and how do you make that happen and all the challenges there.

My first book was actually a book about a Delphi, which is a rapid application development tool. I was definitely on the rapid application development side there. You're exactly right. The thing about those tools was it allowed- the premise of those tools was it allowed developers who didn't have a lot of experience or weren't great software engineers to crank out mediocre quality code very quickly with the idea that it's not going to be around long.

It doesn't matter if it's not fantastic, but then as always happens, the systems never go away. I remember a bulletin board systems before the internet, and I used to go to this program or bulletin board system. They always had sign off pages. I remember the sign off page that said- and this was in the late '80s- that said, "You may be able to see forever as the C programming language, but COBOL still print your paycheck." That's probably still true now, even in 2021.

Jim: Yes. It was interesting in the late '80s, early '90s, I had been doing work on methodologies for a number of years. I really got to thinking, "Yes, but that's not the way I work." I work in more of an iterative fashion. I happened to get a call from a guy who worked for a big computer company at the time named Sam Bayer. We're still good friends, have been for nearly 30 years. They had a RAD tool on the mainframe that had a very long salecycle because people didn't understand what it could do. Sam and I developed an early RAD methodology iterative and very customer-oriented.

We would go into an organization with a team of three to five people, and we would build an application for them in a month, with one week iterations. That was some of the early RAD work that I had done. It was interesting because we did a project for large banks in New York City.

It was very successful project. We got the software. It was going to deliver value to them. We took it down to Ops. We said, "Could you implement this software for us in operations?" They said, "Yes, it'll take about six months." Again, the slow and the fast really interacting with each other during the '90s.

Neal: Well, that's ultimately what led to the gathering that formed the Agile Manifesto, right? You were one of the people who were doing these iterative software ideas, and a bunch of you gathered together and realized that there were some commonalities there.

Jim: That's right. I had sort of on a periphery known about Scrum. I had known about the SDM from Europe. I had known about a couple of other approaches. It was interesting. When I first met Martin Fowler in 1997 in New Zealand, he and I were both speaking at a conference down there. I really wasn't involved much with the OO crowd as Kent Beck and Martin were.

Eventually, out of that, in the late 1999, something like that, before my book was published and before Kent was published, Kent Beck and I exchanged manuscripts. I got invited to a meeting, kind of the pre-Agile Manifesto meeting with a bunch of other ex-peers to see that we had some common ground.

All of these people that were doing some several different kinds of- at that time, they will call it light methodologies- sort of generated the idea of the Agile Manifesto meeting. Which, by the way, we had no agenda for. We had no objectives. We just got together when we started. We didn't know what was going to come out. Several of the people had no hopes that anything would come out of them.

Rebecca: Well, it was around that time too, if I remember correctly, that Nicholas Carr published the book, basically, IT is relevant. I think part of where that comes from, again, is the disconnect between the speed at which the business wanted to evolve and the speed at which the IT systems at the time could evolve.

It feels like what we're going through now is that the endgame, if you will, of the acknowledgement of the role that technology should be playing in business, and it isn't a cost center anymore. I can't be thought of just as a cost center anymore because that doesn't reflect the reality of how businesses are operating, but that started in the 2000s.

Jim: Right. That article, I mean, basically looked at IT as a commodity product. As a commodity product, it was basically cost oriented. Unfortunately, I think that article which we did in the Harvard Business Review, the first article was, I thought put a damper on IT for a few years. Then what happened in the mid 2000s is you had this technology explosion all at once.

You had social media. You had big data. You had iPhones. You had the internet-driven, and all of a sudden, things just sort of blossom, and so did Agile which accompanied some of those technical things. The needs for business speed it up, too, in terms of the uncertainty of those times. Part of that uncertainty was the technology uncertainty of how these things were going to evolve and what we're going to use it for.

I think that was a big switch. What also began to happen during that period of time is they've switched from internal measurements to external value. What was the value delivered from these Agile systems or from any system. You got away from what I'd call a constraint-driven approach, particularly the projects and product management which was scope schedule and cost, to a value-driven approach which was what's the value of this thing we're delivering. That scope schedule and cost became a constraint on that rather than the goal itself. There was that change happening during the 2000s.

One of the things that happened in the Agile movement and is still happening to some extent is that a lot of the people that were there in the beginning, as it expanded, had some doubts about how the expansion went because they would say that people didn't do this, or they didn't do this. That they were too much involved in the mechanics and not enough in the philosophy of Agile.

I harken back to Jerry Weinberg's law of raspberry jam. The wider you spread it, the thinner it gets. The more widely Agile was put out there, the thinner it got in some organizations. All I can say is it was probably better than nothing at all. Even for some of these organizations who really didn't get it. Hopefully, they have some improvement from Agile, and then there were other organizations that embodied it much more seriously.

Neal: Well, I think this goes back to your original observation in our conversation about the chicken and egg, because every time IT delivers some new capability, business figures out a way to exceed that capability and push it to deliver the next new thing, and then there's this virtuous or sometimes chaotic feedback loop that's constantly going on.

I've always said that as soon as we showed business people that we could produce things like reports, then we immediately became overbooked in terms of what they wanted because they're always going to want more stuff because we just keep delivering more stuff to them. I think to your point, something would have had to have come a long like Agile to be able to support business, but business wouldn't be able to move as fast as it is without something like Agile to underpin that.

Jim: Right. As to which came first, the chicken or the egg, is they evolve together.

Neal: One beget the other.

[laughter]

Jim: The egg hatch the chicken, right?

Neal: You're in the process of retiring. As you're wrapping things up, do you have any observations as you start wandering away from this world and spending less time thinking about [crosstalk]?

Jim: Well, the interesting thing is that all of a sudden, we've been injected with some of the most uncertainty that we've had in a long time because of COVID and the economic and the social changes that have gone on over the last year. Some companies benefit from it greatly, like Amazon, and grocery stores. Some companies suffered a lot, like airlines and restaurants.

This ability to adapt and be more agile I think is going to be a major theme during the next few years, even more so than if we had not had the pandemic. I think a lot of things have changed. They looked back those trends were going, for example, more remote work, but the pandemic just accelerated that very much. I think it also has increased the demand for more digital transformations.

A lot of companies are looking for how they can become more digital. I think that trend has accelerated. It's going to be interesting to see what happens over the next few years as different companies respond to our new situation in different ways.

Neal: Before the pandemic, it was easy to tell companies you have to be wary of unpredictability, but now we have a great example [laughs]. It hit the entire globe about the consequences of unpredictability. To your point, companies can't ignore the possibility now.

Rebecca: The importance of thinking not just about efficiency, but also about resilience. Arguably, one of the things that comes out of thinking about Agile software delivery processes is you are optimizing really more for flexibility as opposed to efficiency. You can get away with fewer people if you lump all of your specialists in their own individual silos and parcel work-out to them, but you don't go very fast in that regard. I think, too, that the importance of adaptability and resilience has really been brought home with the pandemic as well.

Jim: Well, the only thing I would say is I really appreciate the opportunity to go back and look through my own history and how it evolved and to the Agile movement. It's been fun for me to look back and do that reminisce.

Neal: Well, it's a great pleasure just to reminisce too, but also to get your insights because I think I speak for myself and a lot of people that we greatly appreciate your persistence in trying to find a better way to solve all of these process problems and figuring out a way to do iteration and all of your fantastic contribution. Thanks very much, and hope you have a fantastic role in retirement.