Author Q&A: Effective Machine Learning Teams

AI might have captured the world’s attention for its ability to do seemingly complicated things with ease. While the technologies that underpin it — like machine learning (ML) — are becoming increasingly accessible, many ML teams face delays and challenges when bringing these sophisticated models into production. ML product delivery is a team sport that necessitates multi-disciplinary collaboration, ranging from data science to product engineering. 

To help ML practitioners address such challenges, three Thoughtworkers — Ada Leung, David Tan and David Colls — wrote Effective Machine Learning Teams. Published by O’Reilly in March 2024, it’s an essential guide that explains how ML teams – regardless of their size or organizational scale – can reliably and rapidly develop and deliver ML products to meet customer needs. 

Given the relevance of the book to today’s software industry, thanks to an increasing focus on machine learning and AI-driven products and services, we wanted to hear directly from the authors about the subject and the process of bringing their book to life. 

With each of them bringing a unique perspective to writing the book, their answers illustrate the range of their experiences and thinking. Find out what they had to say below…

What’s so hard about building machine learning products?

Richard Gall: What's the hardest thing about building ML products?

Ada Leung: Two things I think are particularly important are breaking down organizational silos and shifting feedback left. While we’ve seen investments towards building ML capabilities, we are also seeing challenges validating — through feedback — that we are indeed solving a customer problem and that the solution is right for people. And, to do it successfully, it really does require us to apply systems thinking; we need to look at how knowledge is structured across the organization.

Dave Colls: Four hard things: Building the right thing, building it right, building it in a way that is right for people, and off-by-one errors.

David Tan: It’s hard to pick one! If I had to, I’d say it’s the silos that some organizations build around data scientists and ML engineers. I’ve seen a small ML team spend months building a pricing ML model for another team only to have them shelve it – essentially thrown away by the other team. If only they had shifted that feedback left, with the techniques that we discuss in our book — product discovery, the right team shape and leadership to foster trust, hypothesis-driven experiments — they would have saved months of effort and morale and ultimately delivered a better outcome for everyone involved. 

Another hard thing is the rapidly changing landscape in AI and Generative AI – e.g. in large language models (LLMs), MLOps platforms, dependency management tools, just to name a few. Before the paint has time to dry – or best practices have time to diffuse and graduate into normative practices – teams just have to run with it and figure it out along the way. 

AI in the engineering process

RG: Can AI help the engineering process? If so, how? And what are the limits?

Dave: Computer-aided approaches have been used in varied engineering disciplines for many decades. AI can help software engineering in different ways: it can help engineers learn technology and plan complex initiatives as well as do coding work, and it can help teams collectively manage their knowledge. The limits currently are in AI’s ability to reason about novel and unique business scenarios (as opposed to pattern-match common historical scenarios). People are going to need to drive those elements of the creative engineering process for a little while longer.

David: As we know, AI-assisted development is handy for specific problems — context-aware code completion, helping you write SQL, writing user stories, for example. We also know that it’s fallible and we still need the human “pilot” to assess the correctness and suitability of AI-generated content. But the main point I’d like to make is that these specific tasks are just one of many parts in the product engineering process. Product engineering is fundamentally a social activity that encompasses information flow between humans: user testing, feature prioritization, system design, and so on. As an industry, we need to educate and remind each other that AI is a helpful tool and not a replacement for these other essential jobs in building digital products.

Ada: Again, what Dave and David said! Successful product engineering includes a human-led creative process, and is very much a social activity. On a human level, having the choice to utilize AI in the engineering process means we are opening up our deep-thinking time on the actual problem at hand. Think about hypothesis-driven experiments, for instance, which can surface outcomes faster because our test and learn (in other words, the feedback loop) is shorter. 

The importance and value of technology books

RG: What do you value in technology books? How would you characterize a really great tech book?

David: I really appreciate it when authors share their first-hand practical experience and not just theoretical knowledge. Sometimes when I’m faced with a new technology or a new problem, I’m thankful to find a resource where someone shares their empirical knowledge, so that I can stand on the shoulders of giants and see steady paths that they’ve trod. 

Dave: Now I think about it, I wish you’d posed this question before we started! I think great tech books have a really simple central idea, but also tremendous depth that keeps you reading beyond the first chapter, gleaning immediately applicable practical advice and new perspectives each time you return to the content.

Ada: Visual aids, relatable analogies (but not too many) that compliment the theory. I find myself reaching for tech books when I’ve spent time learning through doing and experiencing first-hand some of the challenges — of course I’m always in a team so have the support of the team to ‘fail safely’.

RG: Are there any books that informed your approach to Effective Machine Learning Teams?

David: It’s not a book, but the seminal papers “Machine Learning: The High Interest Credit Card of Technical Debt” and “Hidden Technical Debt in Machine Learning Systems” by D. Sculley et al. provided the language and a rubric for thinking about ML systems as a multi-disciplinary undertaking. 

Lean Enterprise was another good one that explains what Lean actually entails, and how practitioners can apply it in their business. I’ve also been deeply influenced by W. Edwards Deming’s 14 points, which feature timeless — but often neglected! — advice, such as: cease dependence on inspection to achieve quality, break down barriers between staff areas, and drive out fear. Many of the practices in our book are extensions and stories of applying Lean in the context of building ML systems.

Dave: Team Topologies was a big influence for me, as evident in Chapter 11 of Effective ML Teams, which I’ve dubbed the “Team Topologies expansion pack for ML”. We do refer to many, many other books and technology thinkers throughout, but some we don’t explicitly reference but are nonetheless big influences for me, and that will broaden any technologist’s perspective, are: Don Norman’s The Design of Everyday Things, Kathryn Schulz’s Being Wrong and Yvon Chouinard’s Let My People Go Surfing.

Ada: For establishing foundational concepts around breaking down work, Jeff Patton’s User Story Mapping has been a good resource. Lean Enterprise and The Design of Everyday Things were also books that have influenced my practice. Marty Cagan’s series of books that talk about product management and leadership are also great reads; Empowered, Inspired and Transformed.

Something to acknowledge — and that we touch on lightly — is the discomfort we experience when navigating ambiguity especially during the early stages of delivery. Brené Brown’s Daring Greatly and her article Courage over Comfort have been helpful in separating between the unknowns that come with the problem itself and how we handle ourselves.

Thanks Dave, Ada and David for taking the time to answer my questions. You can have a sneak peek of Effective Machine Learning Teams here or order a copy from O’Reilly.