Founded in 2015, Frollo is one of Australia's leading open banking providers, trusted by banks, brokers, lenders and other fintechs. They help businesses use open banking to deliver better customer outcomes and experiences across personal finance management and lending. They have also developed a free money management app to help individuals stay on top of and improve their finances.
When Frollo needed to extract a large amount of data to create interactive dashboards, run analytical queries and generate next-level machine learning (ML) algorithms, they approached Thoughtworks to develop and deliver an innovative and future-ready solution.
As a long term Thoughtworks customer partnering on several projects in the past, Frollo knew that Thoughtworks' team of data engineering experts would be the perfect fit to design and develop a solution that would support their overall vision; something they knew they needed but weren’t entirely sure how to define such a solution.
Creating a compliant data platform
The original project scope involved Thoughtworks assisting Frollo in extracting large amounts of data and creating a data lake. After ongoing review and refining requirements, the overall goal became clearer, propelling Thoughtworks and Frollo to embark on an exciting proof of concept (POC) project.
As one of the major holders of open banking data, Frollo was on the hunt for a data analytics platform to formulate insights as quickly as possible. As an accredited data recipient, the solution also had to comply with the open banking Consumer Data Rights (CDR) rules, carrying a stringent set of legal obligations, IT and other compliance requirements.
One of the most significant challenges was extracting Frollo’s data from the current Amazon Aurora PostgreSQL cluster and creating interactive dashboards. In addition, Frollo’s vision was to create a data lake not only to create analytical queries but also to leverage the data to produce machine learning algorithms.
Key challenges:
Ensure adherence to open banking regulatory compliance rules
Required functional, interactive dashboards to query and interpret data
Required data to be as close to real-time as possible
Ability to deliver initial use cases in a short timeframe with a goal to continually expand additional use cases in future
Solution needed to be managed and maintained in-house in future
Achieving an ambitious machine learning vision using Databricks
Frollo approached Thoughtworks with a business use case to create a set of dashboards showing user aggregated data and common transaction patterns. The following requirements were part of the use case:
For compliance with CDR rules, data must be deleted from the data lake within 24 hours when consent is revoked
User access controls into the data lake / access restrictions for least privileged access
Data extraction in incremental mode instead of scheduled full load batches
Automatic schema evolution when data columns change in their source database
To identify these requirements, Thoughtworks facilitated two in-depth workshops with key stakeholders from Frollo. Upon identifying essential requirements, Thoughtworks proposed to create a proof of concept solution showcasing a data lakehouse using Databricks powered by AWS. With its unique Unified Data Platform, Databricks was a natural choice to support Frollo’s analytical prerequisites and machine learning vision.
To ensure continuity, Thoughtworks prepared detailed documentation about Databricks and the solution, including step-by-step guides on managing and adding new analytical parameters in the future. A series of Thoughtworks-led training sessions accompanied this documentation to equip the Frollo team on how to use Databricks, manage workflows to maintain ETL (extract, transform, load) pipelines and how to create new queries in the future that also support machine learning.
Business Outcomes
The entire project took Thoughtworks less than six weeks from inception to completion.
Outcomes:
Secures ongoing compliance with open banking regulations
Ability to interpret data quicker, introducing valuable insights with the use of automated dashboards
Near real-time data queries made possible by applying incremental loads of data
Frollo team trained and upskilled in Databricks
Data now in a format conducive to future machine learning capabilities
