In data engineering we're seeing a bewildering choice of tools and technologies. For less experienced engineers especially, it can make sense to work with an abstraction layer to get into the tools, to focus on the task at hand without having to learn several technology-specific APIs and to have the option of switching underlying technologies without too much effort. Fugue is such an abstraction layer. It provides a unified interface for distributed computing, which makes it possible to run Python, pandas and SQL code on Spark, Dask, Ray and DuckDB with minimal rewrites. However, if your team has already decided on a set of technologies, and if they're familiar with their APIs and deep into tweaking and tuning their backend systems, such an abstraction layer provides less value in our experience.
