Implementing Databricks for Data Science and Engineering Teams: Part 1
The analytics arm of a large midwestern U.S. insurance firm needed to analyze its current state architecture used by its teams of data engineers and data scientists. The company asked SPR to help by focusing on the portion of its architecture making use of Azure HDInsight, and building a proof of concept (PoC) on Azure Databricks Unified Analytics Platform for the company’s future state architecture.
The company had 3 key criteria for implementation of the PoC:
- Focus on use of the R language for machine learning (ML) models
- Account for current data science team development processes
- Preference for use of Azure Data Factory for data pipelines, which the current data engineering team was already looking to adopt
Using these criteria, SPR first worked with the data science team to understand current development processes and to define the PoC scope.
This post is for subscribers only
Already have an account? Sign in.