Google Cloud Dataflow 是一个基于云平台的数据处理服务，适用于批量处理和实时流数据处理的应用。我们团队正在使用 Dataflow 来创建用于集成、准备和分析大数据集的数据处理流水线，在这之上使用 Apache Beam 的统一编程模型来方便管理。我们在2018年首次介绍了 Dataflow，它的稳定性、性能和丰富的功能让我们有信心在这一次的技术雷达中将它移动到试验环。
Google Cloud Dataflow is useful in traditional ETL scenarios for reading data from a source, transforming it and then storing it to a sink, with configurations and scaling being managed by dataflow. Dataflow supports Java, Python and Scala and provides wrappers for connections to various types of data sources. However, the current version won’t let you add additional libraries, which may make it unsuitable for certain data manipulations. You also can’t change the dataflow DAG dynamically. Hence, if your ETL has conditional execution flows based on parameters, you may not be able to use dataflow without workarounds.
