Any software system needs to properly represent the given domain in which it is employed and should always be informed by key aims and goals. Machine learning (ML) projects are no different. Feature engineering is a crucial aspect of engineering and designing ML software systems. Feature Store is a related architectural concept to facilitate the identification, discovery and monitoring of the features pertinent to the given domain or business problem. Implementing this concept involves a combination of architectural design, data engineering and infrastructure management to create a scalable, efficient and reliable ML system. From a tooling perspective, you can find open-source and fully managed platforms, but they're only one part of this concept. In the end-to-end design of ML systems, implementing a feature store enables the following capabilities: the ability to (1) define the right features; (2) enhance reusability and make features consistently available regardless of the type of model, which also includes setting up the feature engineering pipelines that curate data as described in the feature store; (3) enable feature discovery and (4) enable feature serving. Our teams leverage feature stores in production to reap these benefits for end-to-end ML systems.
Feature Store is an ML-specific data platform that addresses some of the key challenges we face today in feature engineering with three fundamental capabilities: (1) it uses managed data pipelines to remove struggles with pipelines as new data arrives; (2) catalogs and stores feature data to promote discoverability and collaboration of features across models; and (3) consistently serves feature data during training and interference.
Since Uber revealed their Michelangelo platform, many organizations and startups have built their own versions of a feature store; examples include Hopsworks, Feast and Tecton. We see potential in Feature Store and recommend you carefully assess it.
