Implementing Databricks for Data Science and Engineering Teams: Part 2
The analytics arm of a large midwestern U.S. insurance firm asked SPR to implement a Pilot phase as a follow-up to the initial proof of concept (PoC) phase that SPR had carried out during prior months. As discussed in part 1 of this two part series, the company had asked SPR to help by focusing on the portion of its architecture making use of Azure HDInsight, and building a PoC on Azure Databricks Unified Analytics Platform for the company's future state architecture.
While we took into account how Databricks usage might fit as part of the targeted production environment, production machine learning (ML) models and data workloads were not included in this initial effort due to the sensitive nature these were seen to have at the time. As such, following the conclusion of our work on the PoC phase a separate Pilot phase was carried out by SPR specifically to perform the following:
- Migration of 3 production ML models (ideally with no changes to the original code)
- Migration of production data used by these models
- Recommendations based on what we saw during the Pilot phase
Using these directives, SPR first built out the components used in the initial PoC, albeit using the firm's existing Azure accounts.