IT delivery in unusual circumstances

Podcast transcript

Mike Mason:

Hi, everyone, and welcome to the podcast. My name is Mike Mason. I'm one of your regular hosts, and I'm joined this week by Pat Sarnacke, who is a delivery principal at Thoughtworks. Thanks for joining us, Pat.

Patrick Sarnacke:

Thanks, Mike. I'm happy to be here.

Mike Mason:

And today we're going to talk a little bit about Pat's Thoughtworks journey. So for anyone who's interested, the two of us have just spent half an hour running around trying to find a microphone that works, and eventually turning off all the other electronic devices in the room, including my Bluetooth headphones, which were finally the cause of the problem. So two of us as a couple of reasonably technical guys are feeling frustrated by tech right now.

Patrick Sarnacke:

Oh, I'm not an RF engineer, so I don't feel too bad about it.

Mike Mason:

We kept spotting more things in the room, that could be emitting RF noise and then finally fixed all of that. So, Pat and I have known each other for a long time now. I've been at Thoughtworks for 16 years. How long have you been here?

Patrick Sarnacke:

Over 22.

Mike Mason:

Over 22. Right, so I'm a Thoughtworks baby compared to Pat.

Patrick Sarnacke:

You are, you are. Yeah.

Mike Mason:

Where did we first meet? In the UK?

Patrick Sarnacke:

No, I think you came in here for an Away Day, and I think we met on one of those Away Days.

Mike Mason:

Oh, okay. So an Away Day is a Thoughtworks annual tradition of getting together and fixing all of the things that are wrong with the world, and it's kind of like a mini internal conference, and we do a lot of that kind of thing. It's kind of important to our culture. So I'm not surprised that I met you at one of those. So, Pat, you're a delivery principal at Thoughtworks, which is what you do this week but can you tell us a little bit about what you're going to do next week?

Patrick Sarnacke:

Sure. So just recently Thoughtworks has asked me to step in and help out with our Ecuador office, and so I will be moving to Ecuador in, I don't know, a couple of weeks here to take over the managing director position.

Mike Mason:

That's awesome.

Patrick Sarnacke:

Yeah, I'm pretty excited about it. A little bit nervous but pretty excited.

Mike Mason:

So maybe we can talk a bit about your career trajectory at Thoughtworks. So 22 years at Thoughtworks, delivery principal, what does delivery principal mean?

Patrick Sarnacke:

Sure, it can be a difficult role to describe. I often when I talk to clients about it, I say it's sort of a, it's an account manager focused on the delivery aspect. Making sure that we provide to our clients the things that we're going to promise or the things that we have promised to our clients, making sure that they're happy and that our people are happy, and that we get the work done.

Mike Mason:

Right, right. So it's a kind of a management role but do you have a technical background? Or are you... Tell us about that?

Patrick Sarnacke:

Sure. I do. Not every delivery principal does. A lot of them come up through the project management ranks, and don't necessarily but I started out in our TechOps department. In fact, I was the only full time TechOps person when I joined. There were 45 people at Thoughtworks and so I set up all the laptops, I fix the phone system, I configure the routers, manage Lotus Notes.

Mike Mason:

Oh, Lotus Notes. Oh, fun memories of Lotus Notes.

Patrick Sarnacke:

One of those statements is a lie. Yeah, and then I moved on from that eventually into development, and I was a Java and Ruby developer.

Mike Mason:

Cool.

Patrick Sarnacke:

So yeah, I have a bit of a technical background.

Mike Mason:

So you have a bit of a technical background and the delivery principal role is about getting projects over the line, making sure that we meet our client commitments, corralling some high ability developers who can also be, I don't know, quite demanding, and you know?

Patrick Sarnacke:

Yes, absolutely, and it's usually from the project management side, more so than the technical side. We usually have a technical principal that handles the technical excellence.

Mike Mason:

Excellent.

Patrick Sarnacke:

And I tend to handle the rest of it.

Mike Mason:

Right, right. Cool. So I know that you're the kind of person I've run into around the world. I mean, certainly early on in your TechOps role, going from a 45 person company where you're sort of doing the laptops and the phones to hundreds of people kind of global company. That's got to involve for you as one of the IT guys a lot of kind of global travel. I feel like you're one of my exotic friends who I meet in interesting places, Istanbul being one of them. That was an amazing place to visit. I mean, tell us about some of the countries you've been to.

Patrick Sarnacke:

Sure, sure. So not counting the US, I have lived in seven countries. Ecuador will be my eighth, and I mean, I always go for work and usually it's on a client project. So I don't know, Istanbul was one of the more recent ones. I was there during the attempted coup.

Mike Mason:

Oh, wow.

Patrick Sarnacke:

And that was a pretty unique experience.

Mike Mason:

Okay, tell us more about that because I sort of remember that. I remember being in a meeting there being very impressed by the city because I'd never been to Istanbul before. But at some point, we could see out the window something had set on fire across the street, and a puff of black smoke came up and most people kind of calmly didn't do anything and I was sort of a little bit freaking out because of the political situation but I guess because everybody else was calm you kind of had to go with the flow there, I don't know.

Patrick Sarnacke:

Yeah, yeah. I mean it's, this happens to people everywhere in a region that's suffering instability. At some point you just have to get on with life, and I have to imagine this is something like the UK and Ireland where during The Troubles, at some point you can't just hide in your house you have to go to school, you have to go to work. And so yeah, we did have a bit, I wouldn't say a casual attitude. That's the wrong sentence. But we got on with our life and we couldn't worry too much about the bombings that were happening, and I don't know if we should put this in but certainly I got a great apartment because all the tourists were staying away from Istanbul but yeah, the coup specifically that was interesting because it was a Friday night and I was home, and my apartment was in Galata, which is a really happening neighborhood at the south end of this Istiklal Street, and it's very touristy and there's a lot of restaurants and bars.

Patrick Sarnacke:

It's a very lively place, and I was messing around on my computer and I got a ping from a former colleague, Jeff Gray, who I know you know and he said, "Are you okay?" "Yeah, how are you?" And Jeff, let me know that there was a coup in progress, and sure enough, I went out on my on terrace, and it was dead silent.

Mike Mason:

Wow.

Patrick Sarnacke:

And it's never like that on a Friday night in Galata. But yeah, and then you could hear planes and helicopters flying over and a couple of times, you could hear distant gunfire but mainly my neighborhood was just eerily quiet and the next day as well.

Mike Mason:

And it was an attempted coup, right?

Patrick Sarnacke:

It was an attempted coup.

Mike Mason:

Yeah, okay.

Patrick Sarnacke:

Yeah. In the end. President Erdogan managed to rally his supporters using FaceTime.

Mike Mason:

Wow.

Patrick Sarnacke:

That was the access he had to a nationwide network, and yeah, he rallied people and told them to go out into the streets and fight against the coup, and that's what they did.

Mike Mason:

Okay, all right. And I mean, some of this is, so you've lived in, it's going to be eight countries soon, right?

Patrick Sarnacke:

Yes.

Mike Mason:

And how much of that is driven by kind of your own personal desire to kind of get out and see the world and how much of that is because Thoughtworks needed you to because we've been on this sort of fairly rapid geographic expansion? Is that you personally or the company or like?

Patrick Sarnacke:

I mean, I think it's a happy conjunction, I don't know, of desires. I absolutely do enjoy living overseas. I enjoy travel in general, and it's really great to actually get to spend some real time, get to know people, get to know the customs and really understand how life at least at a superficial level, understand how life works in a new place but you're also right. When you're growing rapidly or you're opening a new office, it helps to have an experienced hand who sort of understands the culture, who understands the business model, can coach people and do training and that sort of thing, and my desire to travel is very convenient because I can go to these places and help build a culture. And I've started, I don't know, I was trying to count one day. I think it depends on how you count but four to five new offices for Thoughtworks,

Mike Mason:

Wow.

Patrick Sarnacke:

Yeah.

Mike Mason:

Wow.

Patrick Sarnacke:

So that's been pretty fun.

Mike Mason:

So okay, let's go through the list then if it's-

Patrick Sarnacke:

Sure.

Mike Mason:

... So you know the countries, right? You said seven or eight. Let's talk about that.

Patrick Sarnacke:

Sure. So the countries and I have to do it in chronological order so I don't forget one, but it's India, Australia, Canada, Brazil, China, Turkey, and Chile. I think that covers all of them.

Mike Mason:

Wow.

Patrick Sarnacke:

Yeah, and then Ecuador will be the number eight. I actually didn't count that on my hands. I hope that was the right number, and some of those have been multiple times. Like Brazil, I helped start both the Porto Alegre office and the Belo Horizonte office.

Mike Mason:

Okay. So I mean, fondest memory or a fond memory? Maybe fondest is asking for a lot.

Patrick Sarnacke:

I mean, yeah, fondest memories is extremely hard. You mean with one of the overseas?

Mike Mason:

Yeah, yeah.

Patrick Sarnacke:

I mean, some of my fond memories, I remember starting Porto Alegre was the first office in a foreign country that I helped start, and it was the first office we had in Thoughtworks Brazil. So we were brand new to the country, reasonably new to opening new offices, and we were just figuring stuff out as we went along, and one of the things that we hadn't really figured out is getting a bank account early, and so we didn't have our bank account yet. It was always almost there but we hired people, and we had to pay them, and we were terrified we weren't going to have a bank account before we had to do payroll. And so at some point, Jen Stille who was the administrative person who helped found the office, walked, went around to all of the expats and asked us to take all the money we could out of the ATM.

Mike Mason:

Oh, wow.

Patrick Sarnacke:

Because we might have to pay cash to our employees for their first payroll.

Mike Mason:

Wow.

Patrick Sarnacke:

So that was...

Mike Mason:

Now is this the same because I've been at Thoughtworks for a while so you hear a lot of stories and it's unsure whether certain things are true or legend or embellished or whatever. But I did hear one story about one of our, I'd call him the sales guy. I think he did a lot more than that for the company, but where he was actually seriously preparing to bring a suitcase full of cash down to Brazil to help with that situation and I think we got the bank account just in time and so therefore, the operation suitcase of cash was aborted at the last minute. Is that true or made up?

Patrick Sarnacke:

I believe it's true. I don't know for sure but I suspect it's the same sales guy who in order to get around the high Brazilian import taxes, was going to smuggle a few Cisco routers in.

Mike Mason:

Oh, gosh.

Patrick Sarnacke:

And we talked him out of it but that was certainly one of his plans. So the suitcase full of cash, doesn't sound out of character.

Mike Mason:

Doesn't sound out of character, all right.

Patrick Sarnacke:

Yes.

Mike Mason:

I mean, it's interesting, right? Because I think as a company, we have gotten better at opening new offices, but certainly back then we would have been in, I don't know, a handful of mainstream, I'm going to call it mainstream countries and say that Brazil at the time for us at least is a newer thing to be doing. So there's learning the language, there's understanding local laws, certainly employment and kind of labor laws in Brazil I think is significantly different than we'd been used to elsewhere. So there's a whole bunch of things for us to learn, yeah.

Patrick Sarnacke:

Yeah, and that's true. I mean, we did have experience. Like setting up Thoughtworks India was a trip. I was always here in the US. I did our first employee in Thoughtworks India is now the CTO of an Indonesian delivery company but the telephone connections were so bad to India at that time that I actually interviewed him over IRC.

Mike Mason:

Oh, wow.

Patrick Sarnacke:

Which was maybe the only time that's happened in Thoughtworks history. But we had a ton to figure out there, and I remember one time the internet went down, and we couldn't figure out why, and it was because the telephone company was doing some work outside and overnight, somebody had stolen all the cables for the copper inside.

Mike Mason:

Oh, right.

Patrick Sarnacke:

And we had those sorts of interesting growing pains in other countries, but China and India, we were quite far removed from Brazil and a lot of that, I don't know how much institutional memory we had around those, but certainly a lot of it had faded by the time we opened up Brazil.

Mike Mason:

Right, right, and so maybe we can talk a bit about the evolution of your role as well, right? So starting out kind of TechOps, which is internal IT at Thoughtworks. Going from TechOps from a sort of, mostly North American based company to a growing company opening offices. Tell me about the transition from that to being a Java and Ruby dev.

Patrick Sarnacke:

Sure. I mean, I guess it might be useful to mention that my degree out of university was General Studies. Okay, so I had some classes in computer science. I had some classes in computer information systems, which was in the B school. I don't think it's really a degree anymore. I don't think anybody does this.

Mike Mason:

The B school, I don't...

Patrick Sarnacke:

Sorry, the business school.

Mike Mason:

Okay, because I was thinking bees apiary.

Patrick Sarnacke:

Yeah, no.

Mike Mason:

Bees, no okay.

Patrick Sarnacke:

We weren't making honey. I mean, not in an academic sense. Yeah, so I had this General Studies degree. So I did have some comp science and programming classes in university, and I got hired in TechOps because my part time job was as a level one tech support guy for a company in Detroit that's now been acquired and he moved on. So I got to, this is probably 2004, and I was starting to get bored, and the company had been about the same size. Somewhere between 250 and 350 people for a year or two, and I didn't see a lot of prospect for growth.

Mike Mason:

For you personally as someone in the TechOps team were you like?

Patrick Sarnacke:

Well, I also because Thoughtworks had been that size for a while. I didn't see Thoughtworks growing. It turns out I have a long history of not predicting good things for Thoughtworks business. I thought Thoughtworks China was going to be a terrible idea. I think I thought Thoughtworks India was going to be a terrible idea. So I'm the wrong guy to predict things for Thoughtworks. But yeah, so my personal growth was also I felt it was plateauing, and I had actually started interviewing outside a little bit, and I didn't really want to leave Thoughtworks but if that was what was necessary for career growth, I was going to do it.

Patrick Sarnacke:

I talked to our founder, Roy at one of these Away Days, we talked about and I mentioned this to him and I just said, "Roy, I think, I either need to move to consulting or I need to move out." And Roy without hesitating said, "Well, move to consulting. It's done." And that's actually one of the things I've loved about working here is people are generally have been very accommodating to career changes and different career goals. That's been really great. But so yeah, so I switched over to the consulting side.

Patrick Sarnacke:

My first project was actually, I guess what I would call technical analysis now. So I was putting together documentation. We were on a crazy project where at some point, I was doing wiring diagrams for hotel rooms with crazy stereo components. You were on that project as well.

Mike Mason:

I remember that project. Yes, in upstate New York. That was super exciting actually, that hobby. So was that your first kind of consulting gig with the company?

Patrick Sarnacke:

No, it was my second. My first one was on an insurance company in Cleveland, and for that one I was doing more, I guess we'd call it DevOps type stuff now, but at the time, that wasn't a word. But that one was my second, and then after that I actually got assigned as a junior programmer to a Java project.

Mike Mason:

Okay. Yeah, I mean, that was an interesting, I remember that gig.

Patrick Sarnacke:

Yeah.

Mike Mason:

Yeah.

Patrick Sarnacke:

Everybody that was on that gig remembers that gig. There were so many insane things that happened there.

Mike Mason:

So I've got no idea if this segment will make it but-

Patrick Sarnacke:

Probably not.

Mike Mason:

... But for listeners, I want to say this was 10 or 12 years ago now. So I can't even do the subtraction so I was like 2008 or maybe 2006 actually.

Patrick Sarnacke:

It's actually yeah, was closer to that.

Mike Mason:

Something like 2006 and there was a, I don't know, billionaire guy in upstate New York who lived in a particular town, I'm not going to name the town because I think that becomes too specific. But his plan was to sort of rejuvenate the town by building this destination kind of shopping Disneyland mall destination thing, and it was ambitious. They got tons of innovative companies involved. We were doing stuff that we, I mean, frankly was outside of our sweet spot, like you talked about designing the wiring for hotel rooms. One of the visions was to have this, remember this is like 2006 or something. So this is before IoT was even a phrase, right?

Mike Mason:

The idea was to have these hotel rooms that were fully wired with an iPad or something that controlled them, and a full shopping experience. So I don't know, you could be out and about at your destination resort thing and see something that you were interested in, and rather than going into a store to try it on, you could say I'm interested in that thing, and by the time you got back to your hotel room, that clothing item would be in your hotel room, not only in your size, but maybe one size up and one size down in a variety of colors or something like that. And you could just, it was this magical shopping experience almost, right?

Patrick Sarnacke:

Well, it was the shopping magic wand. They actually had the idea for a wand. You would wave at the thing if you remember?

Mike Mason:

Oh, okay, right. I just remember the tech in the rooms it was like, okay, can we actually talk to any of this pre-IoT home automation stuff, which is doing all these weird things with a serial protocol and all this kind of business and then TVs with IR blasters and goodness knows what to switch them on and off, and I don't know.

Patrick Sarnacke:

I mean that part of it you mentioned was outside of our sweet spot. Part of the evidence for that is that our approach to it was to create room servers with serial connections to all the appliances whereas if you just hired Lutron or Crestron, they would have got an IR blaster that could handle everything that we were trying to do with an actual server, and then the tablets. There was no iPad at the time. So we had a $7,000 tablet computer that was probably two pounds. Do you remember this at all?

Mike Mason:

I don't know. I don't think I ever saw the tablet.

Patrick Sarnacke:

It was crazy. We wrote a .NET client app for that tablet computer that would talk to the room server over WiFi.

Mike Mason:

And I mean, the funny thing was so while we were doing stuff that was outside of our sweet spot, I feel like most of the time other companies who were also engaged on this thing, were doing stuff outside of their sweet spot. So whoever the architects were, they were being thrown curveballs the whole time on this thing, yeah.

Patrick Sarnacke:

There were some other interesting things there. I wasn't on the shopping project, but we did have people there, and a lot of the stuff they were looking at is now just Amazon Go.

Mike Mason:

Oh, right.

Patrick Sarnacke:

It was 15 years earlier, and the tech wasn't quite ready. We were looking at RFID, right? We're going to tag everything with RFID but then the question was, what if you just got close to the exit of the mall, would it accidentally add it to your account? And that sort of thing. But a lot of the stuff there was the at one point I remember our business analysts came back from a meeting with the designers and was laughing because they wanted a computer just like in Minority Report, where they could move things around virtually and of course now, yeah, we got that. We can DVR but in 2005, that was a crazy thing. We were like, that's a movie, you know that, right?

Mike Mason:

Right.

Patrick Sarnacke:

You can't just buy that. But yeah, they were just ahead of their time in a lot of places.

Mike Mason:

It's interesting because, I don't know, some of these business startup things that you do, you realize that there are no new ideas there're just better executions of them by some people versus others. Okay, so tell us about China and the business continuity plan there because that's kind of topical now.

Patrick Sarnacke:

Sure, sure. So, I worked in Thoughtworks China, I helped start the Chengdu office from...

Mike Mason:

Is that where the pandas are?

Patrick Sarnacke:

That is where the pandas are.

Mike Mason:

Everyone says Chengdu is where the pandas are, so I'm like okay.

Patrick Sarnacke:

Yes, I've been to visit the pandas numerous times.

Mike Mason:

How many?

Patrick Sarnacke:

Sometimes there's a place you take people when you have visitors to your office, and it's really boring. You take them to the same place every time but you know what? It's great to take them to the pandas every time. I'm cool with that.

Mike Mason:

Nice.

Patrick Sarnacke:

Yeah, so I was there from 2012 to 2013, working for an Australian client that was in the insurance financial services industry, and so they had a reasonably robust risk and compliance arm, and one of the things they asked us to do is put together a business continuity plan, and so Joanne Molesky, one of the authors of the Lean Enterprise-

Mike Mason:

Oh, yeah, right, yep.

Patrick Sarnacke:

... Helped us put together this risk and compliance or this business continuity plan, and we put it together. You have to ask all kinds of questions like what would you do in the event of this emergency? Who would you notify? How would you do it? How would you ensure that business would keep happening? And so we did. We put together call trees.

Mike Mason:

So wait a minute, this is us as a service provider to that client, responding to their internal requirement that our vendors will have a business continuity plan?

Patrick Sarnacke:

Exactly. If something happens, how are we going to make sure that our clients software project doesn't suffer. And yeah, so we put together plans for calling trees and an emergency committee, and who we would notify first and when and what frequency updates would come in? How we would provide equipment to our people and then make sure that they could keep working? And I was just talking to one of my friends from Thoughtworks China about the corona virus, and she mentioned that they're actually putting into motion, this business continuity plan that the two of us put together with Joanne, what was that? Seven years ago.

Mike Mason:

Seven years ago, yeah.

Patrick Sarnacke:

Which is, I mean, it's a little gratifying but it's also a little sad.

Mike Mason:

Yeah, I mean, it's a crummy situation.

Patrick Sarnacke:

It's a crummy situation. It was an interesting scenario, I've never had to do this before. But one of the things we had to do is actually go in front of the risk and compliance team at the client and they gave us a fake scenario, it was almost like that Parks and Rec episode if you watch Parks and Rec, if you don't, then nevermind.

Mike Mason:

I'm sorry. Pretend I do. Pretend I do.

Patrick Sarnacke:

No, but they basically gave us a fake scenario, and we had to go through our plan to react to the scenario, and we were terrified they were going to give us virus.

Mike Mason:

Really?

Patrick Sarnacke:

Because that's the hardest one, because we're a consulting firm. All we have is our people and if people start getting sick or getting quarantined and if you remember from the SARS outbreak in, what was that 2009? I don't know.

Mike Mason:

Something like that, yeah.

Patrick Sarnacke:

The government was quarantining people without notice and cell phones weren't as ubiquitous at the time so people couldn't notify that they've been quarantine and so people would just disappear.

Mike Mason:

Like people were quarantined off the street?

Patrick Sarnacke:

Well, I mean, I'm sure they went to a hospital or something. But yes.

Mike Mason:

Right, but they walked on their way to work and kind of got...

Patrick Sarnacke:

Yeah.

Mike Mason:

Yeah, wow.

Patrick Sarnacke:

So we were terrified. When there's an earthquake or a flood, we can rent equipment, we can rent a space, we're confident we can get everybody back up and working from home maybe within a couple of days. That's not a big deal to recover from but when the people themselves are the ones that are getting sick and getting pulled out of commission, that's really hard to recover from, and so we were terrified that's the scenario they were going to give us, and instead they gave us flooding. No problem.

Mike Mason:

No problem. Planning is easy, okay.

Patrick Sarnacke:

We got this. Especially we were on the 13th floor. We'd open the door and the water would run out.

Mike Mason:

I mean, speaking of flooding, I remember there's a client in Canada that you and I both worked on, and their server room was in their downtown office block. Have you heard the story?

Patrick Sarnacke:

No. Keep going.

Mike Mason:

Okay. So their server room is in their downtown office block and they run a bunch of commercial hosting. They're an internet company, I don't want to be too specific. So their server room had an equipment malfunction in it, and because it's downtown office block, and because the equipment malfunction was some kind of smoke situation, the sprinkler system went off because it's a downtown office block rather than a, I'm going to use air quotes "real server".

Patrick Sarnacke:

Yeah, makes sense.

Mike Mason:

Those things, data center thing which would have halon gas or whatever you need. So this was a sprinkler system that went off because of the equipment malfunction, and wrecked all of the equipment in the data center. But fortunately, they had a backup data center. The backup data center was on the floor below the primary data center. So everything seamlessly switched over to the backup data center but then when the primary one flooded, everything went through the ceiling and wrecked the secondary data center and significant services were lost, and take from that what you will but the whole have a backup, test the backup, that's kind of important.

Patrick Sarnacke:

I almost got fired over this sort of thing.

Mike Mason:

Really?

Patrick Sarnacke:

Not water in the server room.

Mike Mason:

No water in the server room, okay.

Patrick Sarnacke:

But yeah, back when I was running TechOps, we used to build most of our own equipment, and we built a RAID array that, I don't know what driver problem there was, I don't know what it was but just never really worked well, and it was on our Lotus Notes server, and at some point, the array failed. We needed to restore from backup and it turned out the backups hadn't worked for three days, and so we lost three days worth of email, and I almost got fired.

Mike Mason:

How did that conversation go?

Patrick Sarnacke:

I mean, so our founder called me and my boss who was the VP of finance into a room, and basically said, I know you guys are trying to do the right thing. I know you're trying to keep costs down but this sort of failure is unacceptable. I've been thinking about firing you. I don't think I will but spend more money.

Mike Mason:

See, I remember a conversation. So I've been part of the TAB, the Technical Advisory Board at Thoughtworks for years now. One of the things that we do is the Thoughtworks Technology Radar twice a year. Look out for that, if you haven't already seen that, and as users of corporate email, obviously, I don't think it's a stretch for someone to claim the Lotus Notes is not high up their list of preferred email systems to use.

Mike Mason:

Obviously, it does a lot more than just email. That's why people install it in the first place. But I remember us having one of these meetings where we're putting together the Radar and you get a bunch of technologists in the room, and we all get kind of adamant about our technology choices and the pitchforks come out and all that kind of thing, and we said we were going to storm the server room and trash the Lotus Notes server but then somebody else somebody pointed out that that's kind of useless because the database is replicated, and we'd be like running around the world trying to nuke Notes databases and they would replicate around and be quicker than we could trash them.

Patrick Sarnacke:

As a former TechOps guy, that was always my argument about Fight Club. Sure they blew up the building where the servers were, but any good financial industry is going to have a disaster recovery site. Nobody's record got lost.

Mike Mason:

Yeah.

Patrick Sarnacke:

Lotus Notes.

Mike Mason:

Lotus Notes.

Patrick Sarnacke:

It took us a long time to get off of Lotus Notes. I think we retired our last note server just a couple years ago, right?

Mike Mason:

Right.

Patrick Sarnacke:

And part of the reason for that was we built a lot of our internal apps on Lotus Notes, and the reason for that is because when I was running TechOps if I had developed anything in Java, my developers would have been staffed. Anytime we had a client, we would have staffed them, and so the only thing I could guarantee I wouldn't lose my development team is if I built in a technology we didn't sell, which was Lotus Notes.

Mike Mason:

I had never put two and two together on that.

Patrick Sarnacke:

That's why our stuff was in Lotus Notes.

Mike Mason:

We did occasionally get client requests for notes stuff.

Patrick Sarnacke:

We did. But we usually said no. Occasionally, I had to outsource my people to a client.

Mike Mason:

Okay, so continuing the kind of the career rock that we're talking about, where have we gotten to so far?

Patrick Sarnacke:

So, yeah, we basically got hung up before I started coding, or just when I started coding.

Mike Mason:

Okay, so yeah, tell me more about this. So, Thoughtworks technologists are not unimposing people, right? And so you've been at the company for a long time, but you go from actually probably a fairly senior guy in TechOps, right?

Patrick Sarnacke:

Mm-hmm (affirmative).

Mike Mason:

To I think you said junior developer.

Patrick Sarnacke:

Yeah, I mean I was definitely the [crosstalk 00:35:13].

Mike Mason:

And a mid career change, right? Tell me about that.

Patrick Sarnacke:

No, I mean, I don't know that it was a problem for me. It was such a strong desire to move into professional services and to start along this career path, that it really didn't bother me to go from being the boss of the thing to being the guy who writes unit tests for the code that somebody didn't test properly, right?

Mike Mason:

Right.

Patrick Sarnacke:

And fixing the junky bugs. If you remember, there's a couple of ex-Thoughtworkers, but they wrote a book that I believe it ended up being called Apprenticeship Patterns, and one of their patterns was sweep the floors. It's when you're the first guy, when you're the most junior Guy, you do anything you can to help the team, and I was extremely happy to be doing that. So that was not a big deal and my team, they were super patient, they were really helpful. I still owe Ji Wang a big debt for coaching me through learning tapestry which was a bit of a stretch for me. But yeah, that was great.

Patrick Sarnacke:

I was able to rely on the few Comp Sci classes I did take in my years of general computer experience to... I certainly never became the most senior person on the team.

Mike Mason:

Here's a thing as well because a mid career change, you're also going to be older and more generally experienced in the world than someone who's kind of closer to just having finished college or they were just coming into the workforce, right? So was that useful? Did that...

Patrick Sarnacke:

I think, I don't feel like it was on this very first project. But my next one, I feel like that actually did help out quite a bit.

Mike Mason:

So did you end up accidentally or otherwise doing management-ish responsibilities, even whilst being a developer? How did that play out? Because one of the things that technologists are always worried about is like, as soon as I start doing non programming things, as soon as someone realizes that I'm good at having a conversation as well as doing the code, then I'm screwed, because all I'm going to do, I'm never going to get keyboard time anymore because suddenly the perception is the developers are not good at talking about what they do I disagree with that, I think whatever. But the perception is there are a few people who can do the talky bits than the programming bits.

Patrick Sarnacke:

Yeah, I mean, on this first project, the project was remote. So our team was in the Chicago office and the client was elsewhere, and so there wasn't as much need for that sort of client facing talky bits. But yeah, I think on this first project, I was really heads down just trying to contribute and learn as much as I could. My second project though, there were four of us plus a project manager, so I guess five of us that showed up at a hosting company to help them out, and it was Ruby, and it was my first Ruby project.

Patrick Sarnacke:

So I was by far the least experienced person but I had domain expertise because it was a hosting company and I used to run TechOps, and so my expertise in that domain, in networking and it was especially IP addressing, because I could do all those in my head from my long years running our networks here. That combined with my experience in my general communications and people skills, the clients used to actually use me as sort of a check on the team to see if I thought something was a little crazy they'd pay more attention because I was the mature guy in the room, and you can never be sure that those younger developers aren't going to go off and over engineer or gold plate or do something crazy.

Mike Mason:

Sure, right.

Patrick Sarnacke:

So that was fun. My favorite pairing experience is actually not a great one. There was a guy on the team that was just way better at Ruby than me. He was a way better coder. He's probably much smarter than I am, and I hated pairing with him because he just used to leave me in the dust and I would lose track of what's going on, and he would never stop to explain. He was not great at pair programming.

Mike Mason:

It's not a great advertisement for the experience of pairing. So, okay. Yeah.

Patrick Sarnacke:

He was not great at pair programming, and so, I was pairing with him one day, but then we got to a card that was about, we were managing their IP address space, which if you don't know, IP address space is handed out only as necessary, because it's a limited space, at least IPv4, and it turns out, they'd been losing space. They'd been subnetting, and then they forgot they had a whole chunk of addresses, and then I believe it's ARIN is the group that hands out the space, and ARIN would yell at them for requesting new space when they had existing space.

Mike Mason:

Wow.

Patrick Sarnacke:

And so they wanted to track their space, and so we're writing an app to help them do that, and we got to a thing where we need to do to subnet an IP range and figure out what their remaining subnets were, and this guy took the keyboard and was banging away furiously leaving me behind and couldn't solve the problem. And finally, I said, hey, man, can I drive for a second? And I wrote the whole algorithm in 15 minutes because I used to do it in my head all the time, and that was actually one of my proudest days was being useful with this guy that was way more skilled than I was.

Mike Mason:

I had an experience when I moved to Canada, I was pairing with a guy. He's just a great programmer, very brusque, in his delivery of things, and we were coding something, I can't remember what it was, and I was trying to offer a suggestion, and actually this is probably not great pair programming technique either, right? I was offering a suggestion on the exact loop or whatever it was he was writing, right? Which is, there's a recent article on martinfowler.com on pairing a couple of German colleagues have pulled together a bunch of really useful information on how to pair well. And actually, I think, my understanding of pairing certainly at the time was not that great.

Mike Mason:

So I was nitpicking somebody else's code, which is maybe not what you want to do when pairing. So I tried to offer a suggestion, and he kind of brushed me off in this very brusque way that he has, and I just sat back because I could see the way he was going and I knew that in about three steps worth of time he was going to get stuck because I realized why this thing wasn't going to work, and I just sat there, and then in three steps worth of time he realized that, actually, yes, that was a good suggestion that Mike was offering because the guy is brilliant, right? And not that anyone really wants to point score when it comes to developing but you also want to know that you're not useless and you know?

Patrick Sarnacke:

Yeah. You know one of the funny things about working in China was there's culturally there is some reluctance, and I don't want to over generalize here. I want to be careful talking about other people's culture. But there is some reluctance to criticize people that are more experienced or older, and so one of the problems we had early on in Chengdu was that some people would not give that sort of pairing feedback if they were pairing with somebody who was more experienced, no matter how wrong they were, and that was one of the challenges we had to break that habit and that ingrained response and say no, no, when you're pairing, both people have valid things to offer to the pair. That was some work.

Mike Mason:

Yeah, yeah, I can imagine. I remember I visited Korea, and one of our Thoughtworks alumni was working for a large company in Korea and they were doing software, they were trying to figure out how to do software better, and they'd set a small department that was going to be much more agile, and they were doing all the cool stuff. And actually, they'd done a bunch of stuff really, really well. But he said to me, one of the big challenges is that hierarchical nature of the culture where you don't criticize a superior, and that really breaks pairing.

Mike Mason:

I think it does depend exactly how you do bearing because if you have somebody driving and somebody navigating, and the person who's navigating is stepping out of the minutiae of the code, then maybe offering structure feedback and big picture feedback maybe that is somehow less affronting culturally than nitpicking someone's code but, I don't know, I've always I'm a giant micromanager when it comes to stuff like this.

Patrick Sarnacke:

I think it's fine and it was sometime during that project where I realized, you know what? I don't have the concentration for preparing, and I think a bigger part for me actually was that I didn't have the same level of passion for it as many of my colleagues at Thoughtworks, and I was never going to be a world class coder at Thoughtworks.

Mike Mason:

So you realized that I realized that?

Patrick Sarnacke:

I realized that, and that's when I started investigating other career paths within professional services. I wanted to continue to be a consultant but I was like, you know what? Maybe coding isn't going to be my end game. And so at a similar time, I was at ThoughtBoarder, which I believe you've been to ThoughtBoarder as well.

Mike Mason:

I have been to ThoughtBoarder.

Patrick Sarnacke:

I have been to ThoughtBoarder. ThoughtBoarder is an unofficial annual ski trip, and there's also a EuroBoarder for our colleagues in Europe, and this one was having to be in Calgary or outside. It's actually probably like 30 or something.

Mike Mason:

30, I think, yeah.

Patrick Sarnacke:

And there were bunch of people standing around, and they were talking about their experiences in Thoughtworks India, and I thought, man, I've never been to Thoughtworks India. I want to go to Thoughtworks India, and I began looking for ways to get to Thoughtworks India, and the way that came up was teaching Thoughtworks University.

Mike Mason:

Oh, right.

Patrick Sarnacke:

Which at the time was Thoughtworks Bangalore, and so, I put my name in the hat, and I started advocating and as you know, one of the things about Thoughtworks is we're pretty egalitarian but once you've built up some social capital you can nudge things a little bit, and I don't know how many years that was almost 10 years of social capital, I was able to nudge a few things, and sure enough, I got to go to Thoughtworks University as one of the developer instructors.

Mike Mason:

And so for listeners context here, Thoughtworks University is our graduate training program. So usually when people are coming out of college, or university, we hire those folks and we send them on a, it's a six week, eight-week, something like that.

Patrick Sarnacke:

It's down to five now.

Mike Mason:

Down to five now.

Patrick Sarnacke:

It was six at the time.

Mike Mason:

Six of the time. So its six-week immersion training program in India pulls you out of your current cultural situation, throws you in the deep end in India, helps you build connections with other people who are in the program, and then sends you back to whichever Thoughtworks country you came from, kind of ready to work, ready to be a consultant. So it's a graduate training program at Thoughtworks. So that's Thoughtworks University. So you sort of, it's not quite pulling strings, right?

Patrick Sarnacke:

It wasn't.

Mike Mason:

How would you describe it? You clearly state that you've got a desire to do a thing.

Patrick Sarnacke:

I let people know I was interested, and I let them know a couple more times, and one of the things about Thoughtworks University is, it's a commitment. It's a big commitment in a foreign country, and not everybody can do it. So they're often actually looking for people who are willing to go. At the time to be fair, I wasn't aware that finding trainers was a bit of a struggle. So I was probably pushing harder than I needed to, to get there, but I certainly did and I reached out to the guy I knew there was a running Thoughtworks University and I reached out to our resource management team who's in charge of staffing and that's what sort of thing. But I got the call. I got the nod, so.

Mike Mason:

So TechOps guy, Java and Ruby developer, Thoughtworks trainer?

Patrick Sarnacke:

Yes.

Mike Mason:

Okay.

Patrick Sarnacke:

I mean, it's a bit ironic, I guess. I don't know. Where's is Alanis Morissette? That I got there as a developer trainer right at the time I was deciding I didn't have what it takes to make a wonderful Thoughtworks developer. But thankfully, it was a language I knew. It was in Java, and they paired me with the great Paulo Caroli.

Mike Mason:

Oh, wow. See, I've heard of him.

Patrick Sarnacke:

Yes, written excellent books, super strong developer, great friend of mine. And yeah, I don't want to say he carried me, I contributed plenty but it was wonderful to have somebody as strong as Paulo to be leading that track of Thoughtworks University. But we got to teach things like very basic solid principles, right?

Mike Mason:

Mm-hmm (affirmative).

Patrick Sarnacke:

We talked about single responsibility, and we talked about interfaces, Liskov substitution, the very core of how to write object oriented code well, and then we got to do a project sim and teach people what it's like to be on a client site, and that whole experience was one of my career highlights. It was so much fun to be passing on my small amount of accumulated wisdom to fresh, eager new Thoughtworks hires.

Mike Mason:

So the project simulation that you talked about, was that the fun part or was it the whole thing was it was the fun part?

Patrick Sarnacke:

It was the whole thing, and it was a ton of work, and I should say it's different now than it used to be. We've gone through I think we're on the third revision of Thoughtworks University. So it's changed format a bit. At the time, it was mostly lecture, and I shouldn't say lecture. It was like workshops and coding dojos and stuff but these days, it's much more simulation and much less person at the front of the room talking to you. But there was a ton of learning from the coaches, we did a ton of prep but overall, the whole thing was just such a great experience. Yeah, every aspect.

Patrick Sarnacke:

Yeah, I think the attendees preferred the simulation to everything else, and that's part of why we switched to more simulation and less classroom is because people really liked the simulation.

Mike Mason:

Yeah, so your next major switch was kind of away from the technical?

Patrick Sarnacke:

Yeah.

Mike Mason:

Tell me about that.

Patrick Sarnacke:

Yeah, so as I said, I decided I couldn't be world class at programming but I still felt I could be potentially world class at consulting, and so the next step was to move into iteration management and then project management. So my next project was up in Canada at our mutual client in Calgary. I mean, that was just the start. I from there, I moved on. I was iteration manager on a project in Cincinnati, and then ended up on one here based in Chicago, although the client was in Dallas. Did that for a while, and that was a really great project.

Patrick Sarnacke:

That was the first time I was working on an actual B to C, a real consumer facing client or a project, and I did some rough math in my head and I think that that particular website was worth, it generated $8 billion revenue in a year.

Mike Mason:

Wow.

Patrick Sarnacke:

Which is fantastic, right? That's something that really carries some weight. I really liked working on that project, and something people may not know about Thoughtworks is at the 10-year mark, we get a three-month paid sabbatical.

Mike Mason:

Yeah, 12 weeks.

Patrick Sarnacke:

12-week paid sabbatical, and I had put my sabbatical off for a couple of years. But now I was going to take it after this website launched, and I was starting to do planning and I was thinking about where I was going to go in the world. I was going to visit Southeast Asia, and my project manager came to me and said, Pat, our next project is going to be in Thoughtworks Brazil, and we're going to open an office, and I'd like you to go down and be the project manager for that.

Mike Mason:

Wow.

Patrick Sarnacke:

And so I put my sabbatical planning on hold and said, yeah, absolutely. I'll move to Brazil. I've never been to Brazil at this point.

Mike Mason:

But how hard did you have to think about that?

Patrick Sarnacke:

I didn't think about it at all.

Mike Mason:

You didn't think about it, you just Brazil, yes.

Patrick Sarnacke:

Yes, it wasn't even, like he could have said, our next project is in, I don't know, Siberia, and I would have said, okay, let's do it.

Mike Mason:

Because it was opening an office?

Patrick Sarnacke:

Yeah, because it was overseas.

Mike Mason:

Because it was overseas, okay.

Patrick Sarnacke:

It was a chance to go live in a new country and meet new people, and that sort of thing really makes me happy. I mean, the fact that it was Brazil was fantastic. Brazil is a wonderful country, really loved the people I met there, really enjoyed living there but it could have been anywhere else too, and I would have put my sabbatical on hold, and yeah, we talked a little bit about the start of the Brazil office with the...

Mike Mason:

Money, a case, and cash, and all that kinds of, yeah.

Patrick Sarnacke:

Yeah, but that was the first time I helped open an office and I sort of got a reputation that said, I mean, it wasn't only me. In fact, Paulo, my former roommate from Thoughtworks University also helped start the office. He's Brazilian but we had a great team and we really set it up and now it's one of our big regions for Thoughtworks. We have four offices.

Mike Mason:

Four offices, yeah.

Patrick Sarnacke:

Hundreds of people. I don't remember how many now, but that's actually one of my career, I'm proud of the work we did setting up Thoughtworks Brazil.

Mike Mason:

Well, and it was a big bet for the company at the time. I won't go into the specifics, but it was not clear actually the opening Thoughtworks Brazil was the right strategy for us at the time. So there was a fair amount riding on it, I think.

Patrick Sarnacke:

Yeah, and we had some pretty strong requirements to be profitable, or at least break even. Whereas a lot of times when you open a new office, you get a bit of a leeway. You don't have to make a profit for the first couple of years but Thoughtworks Brazil, they said, no, no, you have to at least break even your first year. And so, we cut a lot of corners. Our firewall was some Linux firewall we weren't using one of the Ciscos that most of our offices use, and our office space was actually, it was in the comp side building at one of the universities in Porto Alegre and it was just two tiny rooms and we had so many people crammed into these tiny rooms.

Mike Mason:

I'm not sure I ever saw it at the tiny room state because I remember the university campus but it was at the point where we had a floor, or a half a floor or something like that. So I don't think I saw the early stage.

Patrick Sarnacke:

Yeah, so they opened a, I forgot what they call them now, but they opened a corporate center in the university, and so now our offices are in there, but before they were ready, the buildings hadn't been built yet. They just stashed us in the comp side building, and so on the same floor they also had some people from Dell computer, and what an agile company we use so many sticky notes that at some point, the Dell people on the floor, thought that we worked for 3M and they wanted to come over and see if they could buy some stickies from us. Which was pretty great.

Mike Mason:

But so you got a reputation as the guy who can open offices.

Patrick Sarnacke:

Yeah, I'm a safe pair of hands that can inculcate the culture and I can deliver. So I can help people deliver software properly. So yeah, after that, that led to my time in Thoughtworks China, that led to a second office in Thoughtworks Brazil, and that led to, Turkey wasn't actually a new office, but they wanted some help accelerating their growth, and so they brought me to Turkey for that, and Chile was a brand new office as well. So yeah, it was really great. They trusted me and anytime they had a new office open, they thought, maybe Pat will go help up in the office.

Mike Mason:

Well, because it's a good combination of skills, right? There's the having been around for 22 years now, that's a lot of understanding the corporate culture and how things work. Having been a developer that's understanding a little bit about how we do our particular flavor of development being a teacher at Thoughtworks University, that's able to help people get up to speed.

Patrick Sarnacke:

I think a lot of the individuals we hire are actually quite ready to go but the IT industry in a lot of markets is not where it is in a country like the US or the UK, and so, even if the individuals are great, they don't have the experience working in agile teams, in a more lean fashion in the sort of teamwork environment that we expect in the US, and so I think there's a learning curve even for the most skilled of our hires in those new markets, there's still a learning curve, and very often the industry in those markets lags. It was a long time before... Brazil is in a great spot now. Brazil, the agile movement, and the quality of code and the entrepreneurs, Brazil is really flowering right now but when we really started, Brazil was mainly just like a waterfall shop. They didn't a lot.

Mike Mason:

Yeah, there's a lot of conservatism in the IT industry in general, yeah.

Patrick Sarnacke:

Yeah, absolutely. So yeah, and something we didn't discuss, but between China and my second time in Brazil, I ran our entry level higher program. So I was responsible for hiring all the college grads, I was responsible for managing their first two years of their career, doing their career development, putting together plans, learning plans, and all that sort of thing. So, that experience also made me valuable when it comes to starting new offices. So I've got that HR experience as well.

Mike Mason:

And so opened up lots of countries. Today, this week, you're living in the US, right?

Patrick Sarnacke:

I am. I am for now. Very soon, less than a month. I'm going to pack my two suitcases, my two carry-ons and land in Ecuador, and I've never been before I hear it's lovely.

Mike Mason:

It is lovely. I like it very much. If you like ceviche, it's really good. I'm a big ceviche fan. Mostly because of Ecuador.

Patrick Sarnacke:

I'm not so big on fish, but I like guinea pig. So I'm looking forward to that.

Mike Mason:

Okay, that's no. It is a thing though, yes.

Patrick Sarnacke:

Absolutely.

Mike Mason:

Okay, so do you feel qualified to be managing director of a country? I mean, I guess that's an unfair question that I'm going to ask anyway.

Patrick Sarnacke:

No, no. I mean, one of the things you find when you talk to Thoughtworkers is a lot of us have impostor syndrome, because we're surrounded by brilliant people and it's not just the people at Thoughtworks, our clients are full of... The IT industry is full of brilliant people, and so a lot of Thoughtworkers are just waiting for somebody to figure out that they're unqualified, and so yeah, absolutely. I feel unqualified, and I can tell you though the parts I'm weak at, I've never done finance. I've never had to deal with that before, and so yeah, that part worries me a lot.

Patrick Sarnacke:

I don't know much about marketing. Now, it seems not that hard. But that's probably the Dunning Kruger effect, right?

Mike Mason:

Right, right.

Patrick Sarnacke:

I'm so ignorant about marketing that I actually have no idea how hard it is. On the other hand, the people management part, the delivery part, the technical part, I feel like I've got those under control. So I'm very excited to work with the team down there. They're fantastic, and they know their stuff, they know their market, they know their people. So I think between them and myself will be able to make things work.

Mike Mason:

Yeah, and I think especially the way Ecuador is positioned within Thoughtworks, we use them as a nearshore delivery partner in a lot of cases, and so having someone like you, who's been a delivery principal, who worked for a whole bunch of clients understands what it means to get software across the line. I think that's going to be very useful to our Ecuadorian colleagues and to their operation.

Patrick Sarnacke:

Yeah, I've done so much work distributed. I actually wrote the internal guide and the internal training to doing distributed projects. So that should help.

Mike Mason:

Just a little bit.

Patrick Sarnacke:

Just a little and on top of that, a lot of the work they're doing on there now is platform work, it's delivery infrastructure and API platforms, and my current project has a giant amount of delivery infrastructure and API platforms. So I'll be able to take that actual client experience and bring it down and help educate people and make sure we're growing the right skills, and I'm actually pretty excited about the synergies there.

Mike Mason:

So is there anything that you've not done in your Thoughtworks career that you would like to someday? Not, I mean, obviously, going to be MD of a country is a big deal. So maybe you need to give yourself a bit of time, but what's on your list?

Patrick Sarnacke:

Now, that's a difficult question, and one of my coaches here has this idea of, I think she calls it experience driven career development, which is not about setting goals so much as taking advantage of opportunities, and I tend to follow that path. That's why we made a good coaching pair, and so I generally don't think about oh, that's a thing I really, really want to do instead I look for open doors and I jump through them.

Patrick Sarnacke:

To be honest, I can't think of a big thing that I haven't done that I'd like to. There's certain things that I fantasize about working in Thoughtworks Thailand, but to be honest, I've been to Thailand and it's hot. It is so hot, and the same was Singapore, right? It is so hot. I don't know that I could live there.

Mike Mason:

But the food's amazing, I mean.

Patrick Sarnacke:

Food is fantastic but I'm not sure it's worth it.

Mike Mason:

Because of the heat.

Patrick Sarnacke:

Because it's so hot.

Mike Mason:

Because you're from Chicago, and we're recording this in January.

Patrick Sarnacke:

Yes, this weather is fine, right? I can't take 85 in humid. That's too much. Yeah, and so I thought about a couple new countries I'd like to go into. I would love to see us open Thoughtworks Mexico, that would be fantastic, and then Thoughtworks Vietnam I was pushing for Thoughtworks Vietnam for a while.

Mike Mason:

Oh, wow.

Patrick Sarnacke:

Nobody listens to me. So it didn't happen. But yeah, other than that sort of broader, I would like to find a new place to travel sort of goal. No, I think my career has been incredibly varied, and I'm actually I consider myself very lucky to have had the career I've had. To some degree, that's what happens when you stick around a company for 22 years. People don't realize that, and a lot of our younger people look at me and think, oh man, that amount of travel is amazing.

Patrick Sarnacke:

I didn't go overseas for the first time until I'd been here 10 years, and so I put in my time, I learned a lot of lessons and certainly not everybody has to wait that long but you get more with experience and with, I don't know, loyalty, I guess, for lack of a better word.

Mike Mason:

Right. Well, I think we're pretty much running out of time. You and I have chatted for a good long while here. I'd like to thank you very much for coming on the podcast. Pat Sarnacke, thank you very much.

Patrick Sarnacke:

Absolutely. It's a lot of fun.

Mike Mason:

And thanks for listening. Hope you enjoyed it.

Mike Mason:

Join me on the next episode of the podcast where I'm talking to Gary O'Brien about digital transformation and what that means for the technologist.