Erik on Software

Implementing Databricks for Data Science and Engineering Teams: Part 2

---

This post is the second part of a soon to be published case study I wrote about a recent client project implementing Azure Databricks with my team for data science and engineering teams (the first part can be found here).

---

The analytics arm of a large midwestern U.S. insurance firm asked SPR to implement a Pilot phase as a follow-up to the initial proof of concept (PoC) phase that SPR had carried out during prior months. As discussed in part 1 of this two part series, the company had asked SPR to help by focusing on the portion of its architecture making use of Azure HDInsight, and building a PoC on Azure Databricks Unified Analytics Platform for the company's future state architecture.

While we took into account how Databricks usage might fit as part of the targeted production environment, production machine learning (ML) models and data workloads were not included in this initial effort due to the sensitive nature these were seen to have at the time. As such, following the conclusion of our work on the PoC phase a separate Pilot phase was carried out by SPR specifically to perform the following:

• Migration of 3 production ML models (ideally with no changes to the original code)
• Migration of production data used by these models
• Recommendations based on what we saw during the Pilot phase

Using these directives, SPR first built out the components used in the initial PoC, albeit using the firm's existing Azure accounts.

Background on Databricks Proof of Concept (PoC)

As discussed in part 1 of this two part series, criteria for the PoC was threefold: (1) focus on the R language for ML models, (2) account for current data science team development processes, and (3) preference for use of Azure Data Factory for data pipelines, which the current data engineering was already looking to adopt.

As part of the PoC work, SPR assessed products associated with Databricks and the broader Azure ecosystem, and determined that some were either not production ready or were not an enterprise fit for the company. For example, while MLflow appeared to be a promising open source framework (created by Databricks) to standardize the end-to-end ML life cycle, it was also in alpha and R language support had just been added. Additionally, while we successfully set up Visual Studio 2017 to remotely connect to a Microsoft Machine Learning Server instance via Microsoft R Client, we did not recommend use of this tooling because it did not seem to be positioned well.

After building the PoC, we additionally broke down the types of work that can be done into 12 tasks that we categorized into DevOps and data pipeline work. Components used to build all of this out included Azure Databricks, Azure Blob Storage, Azure Data Factory, RStudio Server, Azure DevOps, R, Python, Databricks CLI, PowerShell, RStudio Desktop, and Git.

Databricks Pilot

We ended up migrating a total of 6 production ML models (2x the initial goal) from HDInsight to Databricks, along with the production data used by these models. All of these models are intended to predict the level of consumer adoption that can be expected for various insurance products, with some having been migrated from SAS a few years prior at which time the company moved to making use of HDInsight.

The strategy that we used for the first 3 models was to migrate a relatively easier model followed by models of increasing complexity, the goal being to get basic migration details worked out first before progressing to others which depend on these basics. Of the 14 specific technical implementation issues we encountered, we resolved or implemented workarounds for all 14, and all of these issues were encountered and surmounted during our work on the first 3 models, leading to significantly increased team velocity for the latter 3 models.

Key challenges along the way can be categorized into 4 areas:

1. poor data quality (biggest challenge)
2. lack of comprehensive versioning
3. tribal knowledge and incomplete documentation
4. immature Databricks documentation and support with respect to R usage

To address these key challenges, we made recommendations in 5 key areas as a follow-up to our implementation work:

1. big picture view
2. automation
3. configuration and versioning
4. separation of data concerns
5. documentation

Since client data, platform, and modeling teams are associated with these models, approaches to solutions and analysis and resolution of issues need to be performed across all three teams. While it is common for silos to form within large organizations, it is especially critical for teams to be in sync with each other with respect to data platform usage, and as such we advised that these teams work together to leverage their existing enterprise architecture group in order to achieve alignment.

We also noticed that while work on CI / CD (continuous integration and delivery) automation had been started, what had been built was largely confined to a portion of the work being performed across these three teams. We advised that end-to-end pipelines be built across artifacts managed by the data, platform, and modeling teams because the work products of individual teams has an impact on the others. During our model migration work, we saw first hand the mistakes that lack of automation was causing, and that manual steps were resulting in added cost.

Wholesale configuration and versioning will aid in fully enabling such automation, reducing errors and supporting repeatable builds and deployments, and as such we included doing so as a separately delineated recommendation. For example, all table schemas, reference data, and R packages used by models need to be versioned as one unit within the context of each model, supporting consistent results every time. Additionally, hard coded values were pervasive in the code base and should be converted to easily maintained configurations.

Code deployed as models contained content not constrained to the actual models, so we advised that data needs be managed as a separate concern to increase visibility of data work, as well as to prevent increased complexity and ripple effect in the code base. Addressing data and schema issues separately from work on models will help prevent diverting from business value added model development. For example, data preparatory work should be performed prior to model execution, as model execution should not fail in production due to data related issues.

Lastly, we commented that current documentation is not comprehensive or centralized, and work to move in this direction would support maintainability as well as to more quickly add members to the data, platform, and modeling teams. For example, how can models as currently deployed be executed from the command line? Which Hive tables does a model use? What is the path / lineage of data as it moves from sources to consumers, including models? And where are the responsibilities of the data, platform, and modeling teams documented?

The Pilot implementations of Databricks that SPR built out included all of these technologies save for Azure Data Factory, as DevOps tasks remained our focus for the duration of the Pilot phase rather than data pipeline tasks. That said, we did put together a high-level architecture document to show the firm how a data pipeline might be tackled following the Pilot phase. In addition to the aforementioned components, which included two subcomponents of Azure DevOps (Azure DevOps Repos and Azure DevOps Pipelines), Databricks and MLflow were also included in this architecture.

Databricks Connect permits connections from IDEs (integrated development environments) and notebook servers to Databricks clusters to run Spark code, and was not released until after the conclusion of the PoC phase, during which time it was in private preview. The status of MLflow, mentioned earlier, progressed to beta during the Pilot phase, and additionally was made available as a managed service called Managed MLflow during the concluding weeks of the Pilot phase. And so as momentum appears to be building up around this framework, we wanted to show the company how it might fit in with the rest of the architecture.

The Result

In addition to migrating 6 models (2x the initial goal) and the associated data used by these models from Apache Hive to Spark SQL (while the data remained in Azure Blob Storage, we also mounted to Databricks File System), our key deliverables also included dozens of annotated Databricks notebooks that we developed using PySpark and SparkR, cluster initialization ("init") scripts that we wrote to be executed on Databricks cluster startup, detailed documentation that explained our work and how to perform it, as well as recommendations as outlined above.

While the PoC and Pilot phases of our work were both successfully executed, subsequent work was not performed due to client budget. That said, our work on these two phases convinced a very skeptical Chief Data Scientist that Databricks is the way to go, and as such indicated during our final presentation to project stakeholders that their next goal following our departure was to migrate their remaining models to Databricks as quickly as possible!

Technologies used during this effort included Azure Databricks, RStudio Server, Azure Blob Storage, Azure HDInsight, Apache Spark, Azure DevOps, R, Python, Databricks CLI, PowerShell, RStudio Desktop, and Git.

Afternote: So Why Databricks?

A brief survey of factors you might find of interest which led to our client's Chief Data Scientist changing his mind from skepticism to adoption.

Databricks just works

At the outset, the Chief Data Scientist stated that they had already tried Databricks, and that it didn't work out for them. Databricks hadn't been made generally available in Azure until recent months, so the question was to what extent they had already worked with the product. It turned out that staff lacked needed skill sets, and didn't have the necessary bandwidth to pick these up, so the Director of Data Engineering brought my team and I on board to address this roadblock.

You might have noticed that the first client directive for the Pilot phase was to migrate a subset of ML models with no changes if at all possible to the original code. One of the primary reasons for not changing the code was to prevent downstream effects from introducing change to the code base. The only programmatic changes we ended up needing to make to the code were to address data quality issues, some of which were due to migrating from Apache Hive to Spark SQL as part of the process.

We rigorously compared model output from HDInsight and Databricks for full production datasets, and found only one output value of dozens which differed, affecting the latter 3 models we migrated. In reviewing these differences with the Chief Data Scientist, it was determined that these were due to changes to Spark function "percentile_approx" over time, as the versions of Spark used by HDInsight and Spark were different.

Note that we used a relative tolerance of 0.000000001 when comparing model output, and when adjusting for more tolerance the differences for this sole column disappeared. Regardless, the minor deviance experienced here was considered by the Chief Data Scientist to be immaterial.

No needed changes to library packaging

While this second part of the case study mentions that we encountered and surmounted 14 specific technical implementation issues during the Pilot phase, keep in mind that we had already become familiar with Databricks by building a PoC, during which time we also addressed specific client obstacles associated with their initial trial runs in prior months.

One of the remaining stumbling blocks for the client team had been the ability to install proprietary R packages that they had developed, in contrast with R packages made publicly available via CRAN (Comprehensive R Archive Network). After demonstrating how to do so via notebooks, we additionally implemented the best practice to move installations of all needed R packages to Databricks cluster initialization ("init") scripts so that these would be available to all cluster nodes used by executing models.

Integration with chosen Git implementation

Databricks did not initially offer out of the box support to specifically address integration with Azure DevOps, a product which also had just recently become generally available. But this lack of support was really a limitation of the notebook user interface, which offered minimal explicit support for Git compatible products.

Since we did not want any unneeded user interface dependencies, we first showed the client team how to programmatically integrate with Azure DevOps, and later also implemented automated syncing of notebooks to Git to minimize the need for developers to do so while programming, as well as to store all of the code into the same centralized location already being used by client teams.

Notebook alternative exists

While the Chief Data Scientist made it clear that modelers on his team use a variety of development tooling, he also made it clear that he was highly averse to using notebooks. While notebooks have quickly become widely popular for data science in recent years, he was accustomed to programming via command line and an IDE called RStudio Desktop, and did not like the idea of being constrained to using notebooks for development, as he found some scenarios (such as for debugging) a bit challenging.

Our solution made use of RStudio Server, the installation of which we included in our cluster initialization ("init") scripts, to provide the option to make use of a browser based version of RStudio Desktop. Also, since the Azure implementation (unlike the AWS implementation) of Databricks does not permit SSH to clusters, RStudio Server additionally provides command line access similarly to RStudio Desktop.

And as the notebooks that my team and I developed intermingled Python to execute pipeline code, R to execute models, and SQL for exploratory work, by the end of the Pilot phase client teams began to see this as an advantageous feature. 

Hadoop goes away

While noting again that this afternote should just be considered a brief survey, several challenges were either expressed by client teams or experienced by my team and I during our migration work from HDInsight to Databricks. As we needed to compare data output from models executed in HDInsight and Databricks, this required us to first execute on HDInsight, and this required our needing to make use of Apache Hadoop YARN for resource management and job scheduling and monitoring.

The HDInsight cluster was being brought up at the beginning of the day and brought down at the end of the day, and Hadoop processes were sometimes unavailable, preventing the ability to resume our testing, and when this issue did not surface we still needed to be concerned with configuration, the improper setting of which resulted in obscure error messages.

After just migrating the first model, the Chief Data Scientist was already expressing satisfaction with not needing to fiddle with YARN etc. Databricks is not built on Hadoop, and therefore does not have the legacy constraints of Hadoop. And since it had been taking hours each day to bring up the HDInsight cluster used by client teams, bringing up Databricks clusters on an as needed basis was seen to be a significant benefit.

Client focus was on the Apache Spark component of Hadoop, so why be encumbered with unneeded legacy constraints, not to mention the much wider available selection of Databricks cluster node configurations.

Personal Reading List: 3Q2019

My personal reading list for July, August, and September 2019.

Color Key: Special Notes, Completed Reading. Updated throughout the Quarter.

---

At Bat (Schnell)

---

Disruptive Analytics: Charting Your Strategy for Next-Generation Business Analytics, by Thomas W. Dinsmore, Apress, 2016.

The Model Thinker: What You Need to Know to Make Data Work for You, by Scott E. Page, Basic Books, 2018.

---

 At Bat (Langsam)

---

AWS Certified Solutions Architect Associate All-in-One Exam Guide (Exam SAA-C01), by Joyjeet Banerjee, McGraw-Hill Education, 2018.

Practical Statistics for Data Scientists: 50 Essential Concepts, by Peter Bruce and Andrew Bruce, O'Reilly, 2017. Copy provided by MarkLogic.

---

 On Deck

---

Thinking Architecturally: Lead Technical Change Within Your Engineering Team, by Nathaniel Schutta, O'Reilly, 2018. Copy provided by Pivotal.

Evolving Data Infrastructure: Tools and Best Practices for Advanced Analytics and AI, by Ben Lorica and Paco Nathan, O'Reilly, 2019. Copy provided by O'Reilly.

---

In the Hole

---

Getting DataOps Right, by Andy Palmer, Michael Stonebraker, Nik Bates-Haus, Liam Cleary, and Mark Marinelli, O'Reilly, 2019. Copy provided by Tamr.

Machine Learning at Enterprise Scale: How Real Practitioners Handle Six Common Challenges, by Piero Cinquegrana and Matheen Raza, O'Reilly, 2019. Copy provided by Qubole.

Amazon re:MARS 2019 Day 2-4 (June 5-7, 2019)

Amazon re:MARS 2019 
Aria Resort & Casino
Day 2-4 (June 5-7, 2019)

---

From the promotional materials:

KEY2: Opening remarks & Innovation Spotlights (Day 2 / 9:00am - 11:30am)

Amazon keynote led by Jeff Wilke, CEO of Worldwide Consumer. Keynote to include Innovation Spotlight talks by: 1) Tom Soderstrom, IT Chief Technology and Innovation Officer, Jet Propulsion Laboratory, 2) Daphne Koller, CEO and Founder, insitro & Rajeev Motwani Professor in the Computer Science Department at Stanford University, 3) Kate Darling, Researcher, MIT Media Lab, and 4) Colin Angle, Chairman, CEO and Founder, iRobot.

---

---

My personal notes:

• Jeff Wilke, CEO of Worldwide Consumer
• some have been working on AI for decades, but we're only just beginning
• Wilke discussed operations and how only one individual named Skip took care of it, and later operations developed algorithms to essentially do an "auto-Skip"
• the first classical ML method used was item-item collaborative filtering, considered revolutionary at the time, e.g. customers who watched such and such movie watched such and such movie
• graph clustering techniques came later, as did deep learning
• state of the art methods ended up performing worse than simple methods
• the first commercial deep learning model was simplistic but performed better and scaled better
• performed 2x better than item-item collaborative filtering
• emphasis on starting with customers first and then working backward
• the expected customer shopping experience is much different than in the past
• a product called StyleSnap, which permits providing photos that help match looks with what Amazon sells
• Dilip Kumar, VP Amazon Go
• there were many tech options early on in terms of how they would implement, but they ended up settling on computer vision
• Kumar commented about some of the challenges they needed to surpass
• one example was the constant influx of new products being available in the store, and they needed to handle this with little data
• the hardest part was making the technology invisible to the customer
• the shopping experience is the same except for one thing: you can just walk out
• Jenny Freshwater
• forecasting, buying, placement, customer promise
• need to account for price elasticity, products that won't sell, and regional demand
• two of the hardest challenges are new product demand and seasonal product demand
• SQRF was the first algorithm they used, based on the random forest algorithm: random forest + map reduce
• they started evolving their algorithm to include deep learning, leading to a 15x improvement in accuracy
• in 2016, they built a feed forward neural network (FNN)
• but FNN also had its limitations: modeling and training would take months each time they entered new markets etc
• convolutional neural network (CNN): demand history for prior products can help predict demand for new products
• unfortunately, these black box models cannot be fully explained like before
• neural networks make impossible forecasting possible
• Brad Porter, VP and Distinguished Engineer, Robotics
• advancements in robotics
• Kiva was purchased in 2012 and became Amazon Robotics
• robotic palletizer: how to get the right inventory to the right placement
• customer expectations keep increasing
• middle model network: connects delivery network with last mile providers such as USPS
• Pegasus system introduced today
• due to their investment in simulation, they were able to get Pegasus ready quickly
• this gave them a chance to rethink robotic drives
• canvas technology will enable these robots to interact with staff beyond what they have currently been thinking
• Rohit Prasad, VP and Head Scientist, Alexa Artificial Intelligence 
• AI pillars to improve customer experience:
  (1) trust
  (2) smarter via self-learning
  (3) learning directly from customers
  (4) proactive
  (5) natural
  (6) transactions to conversations
• Alexa Conversations introduced today: less effort, less code, less training data
• dialogue flow is now automated rather than manual
• anticipating customer's latent goals
• cross-skill action predictor: this will be available to customers soon
• Jeff Wilke, CEO of Worldwide Consumer
• Jeff came back to present
• the highlight of all of these innovations is ML
• democratizing AI will transform society for the better
• broad acceptance of tech only comes with practical application

---

From the promotional materials:

M10: Infor Coleman: Streamlining Enterprise ML Complexities (Day 2 / 1:00pm - 2:00pm)

AI and machine learning can help solve some of the toughest technical problems today. In this session, you’ll learn how AWS capabilities from AI services to Amazon SageMaker can help developers build intelligent applications and systems.

Rick Rider - Sr. Director, Product Management, Infor
Massimo Capoccia - SVP Infor OS, Technology, Infor
Urvashi Chowdhary - Sr. Product Manager, AI Platforms, Amazon Web Services

---

My personal notes:

• TODO: add notes

---

From the promotional materials:

M31-L: Intended Consequences: The Power of Intentional Technology Implementations (Day 2 / 2:15pm - 2:45pm)

Bringing compute power to our palms has been one of the most powerful transitions in recent history. We can be anywhere in the world and stay connected to what’s important in our lives. However, it has also brought unintended consequences, like children losing social skills by having their faces buried in their phones. Artificial intelligence has the opportunity to make an even bigger positive impact on society. Naveen Rao, neuroscientist, corporate vice president, and head of Intel’s AI efforts, will discuss how to think about AI in a way that helps bring about the positive aspects of the technology’s future, and try to avoid the pitfalls.

Naveen Rao - Corporate Vice President and General Manager of the Artificial Intelligence Products Group, Intel

---

My personal notes:

• TODO: add notes

---

From the promotional materials:

A19-L: Unlocking the Creativity of Humans (Day 2 / 2:45pm - 3:15pm)

ABB Chief Digital Officer Guido Jouret believes robots can automate jobs that fit one of the 3 D's – dirty, dangerous, or dull – allowing people to focus on more creative, highly skilled tasks. In this session, he will discuss the future of job automation with robotics and the positive impacts this will have on our society and careers.

Guido Jouret - Chief Digital Officer, ABB, Inc

---

My personal notes:

• TODO: add notes

---

From the promotional materials:

M23: AI from Prototype to Production: Vertical Market Value (Day 2 / 3:30pm - 4:30pm)

Intel will focus not only on AI advances driven by the power of cloud computing, but on the emergence of distributed computing architectures that leverage inference out at the source of the data in the real world, including as hospitals, cities and factory floors.

Jonathan Ballon - Vice President - Internet of Things, Intel, Intel

---

My personal notes:

• TODO: add notes

---

From the promotional materials:

A26-L: The Artificial Intelligence–Powered Pivot to the Future (Day 2 / 4:45pm - 5:15pm)

AI and automation are fueling a new norm of digital disruption and breeding new forms of competition. To stay ahead, businesses must continuously innovate with speed and agility like never before. Yet, the majority of companies are challenged in embracing new AI-fueled business opportunities and calibrating their investments in innovation. Learn the secrets of “Rotation Masters” and how they are using AI to do things differently and drive double-digit growth.

John Matchette - Managing Director, Accenture Applied Intelligence, Accenture

---

My personal notes:

• we're now able to compete on imagination, which couldn't be claimed 5 years ago
• a disrupter in a marketplace is one who unlocks trapped value
• if you learn the pattern, you can use disruption as a tool
• what do you do if faced with disruption? do things differently and do different things
• only about 6% of companies are good at being disrupters
• these 6% are referred to as the "rotation masters"
• Accenture came up with something called the "wise pivot"
• you need to transform the core
• you need to generate cash to do those new things reinvested
• the next thing is that you need to grow the core, something a lot of businesses miss
• you also need to scale the new
• it's a new discipline, you can't move too fast or too slow
• the first conference presenter I actually heard mention "munging" data
• time series and human judgement are 2 ways forecasting used to be done
• Matchette commented that he has a lot of past experience in supply chain
• humans can probably only handle 2 ways, but these restrictions don't exist with ML
• you can use disruption as a competition strategy
• "data driven is hard to do"
• just because you're a competitive engineering company doesn't mean you're going to be a competitive data science company
• Matchette commented that much of this information is outlined at this link, which includes a recently published book by Omar Abbosh

---

From the promotional materials:

KEY3: Thursday Keynote & Innovation Spotlights (Day 3 / 9:00am - 11:30am)

Amazon Web Services keynote led by Dr. Werner Vogels, CTO of Amazon.com. Keynote to include Innovation Spotlights talk by: 1) Andrew Ng, Founder and CEO, Landing AI | Founder and CEO, deeplearning.ai, 2) Aicha Evans, CEO, Zoox, 3) Andrew Lo, The Charles E. and Susan T. Harris Professor, MIT Sloan School of Management | Director, MIT Laboratory for Financial Engineering, and 4) Ken Goldberg, Engineering Professor, UC Berkeley | Chief Scientist, Ambidextrous Robotics. This keynote also includes a conversation with Jeff Bezos, Founder and CEO, Amazon.

---

---

My personal notes:

• TODO: add notes
• Jeff Bezos, Founder and CEO, Amazon
• interview by Jenny Freshwater
• builders and dreamers: the dreamers come first, and the builders get inspired by them
• the dreamers can dream more as the builders build
• the dreamers can't continue to dream until the builders build
• you needs to be right a lot: telling someone to be right a lot isn't super helpful
• people who are right a lot listen a lot
• they also change their mind a lot
• changing minds based on data makes sense, but you can also change your mind without data
• if you don't change your mind a lot, you're going to be wrong a lot
• you need to disconfirm your biases
• it's human to be biased
• you're going to be right more often if you can disconfirm biases
• Bezos would probably have been an extremely happy software engineer if he hadn't done Amazon
• he seeks out people who say no, and has had trouble winning support for his ideas
• disagree and commit is a leadership principal at amazon
• it works in both directions, it's not just the boss who says to agree with them
• you're not going to have the same ground truth as someone else
• it's not a matter of being right sometimes, but a willingness to gamble with someone on something
• predictions about the next 10 years are hard to make
• the thing that's funny though is that nobody goes back and checks to determine accuracy
• robot grasping has turned out to be extremely challenging, which was unexpected compared to vision etc
• the answer to "what's not going to change" provides confidence that putting energy into them today will have staying power, stable in time not going to change
• strategy should be based on customer needs that are stable over time, rather than competitors which dynamically change over time
• a recently taken big bet at Amazon was Project Kuiper: Leo satellite, broadband everywhere
• (a woman rushed on stage to interrupt the interview, it sounded like it had something to do with animal rights)
• in being asked how he knows when to throw in the towel, Bezos said he doesn't like to give up
• when the last high judgement champion has thrown in the towel, that's the time to do so
• if an executive still thinks their's hope, Bezos will hang on and continue
• Blue Origin is targeting the moon because it's the resource rich neighbor of Earth
• water in the form of ice by poles
• moon is close, only 3 days away
• unlimited launch opportunities
• logistically a good place to go
• the reason to go to space is to save the earth
• heavy industry needs to be moved off earth
• operational, reusable launch vehicles is the number one important thing
• infrastructure is always expensive
• it was cheap to start Amazon because it relied on existing transportation networks like USPS and credit cards
• the goal of Blue Origin is to build that infrastructure
• you can't start a space firm from a dorm room like Facebook because the infrastructure is too expensive, the heavy lifting isn't done yet
• advice to entrepreneurs for starting a business: be customer obsessed, how to absolutely delight them
• you need to have passion for the arena in which you're going to work
• you can't be a mercenary, you need to be a missionary
• you have to be willing to take risk
• if there's no risk with a particular business idea, it has probably already been done
• you need to accept the fact that your business is an experiment, and it can fail

---

From the promotional materials:

M11: Predicting Weather to Save Energy Costs (Day 3 / 1:00pm - 2:00pm)

Learn how Kinect Energy Group uses advanced machine learning capabilities to predict electric spot prices for regional power markets using the Amazon SageMaker DeepAR time-series forecasting model, incorporating historical pricing and weather data to drive the machine learning models. Improved price predictions assist with increased trading volumes for forward pricing contracts.

Richard Delisser - Vice President, Global Infrastructure, Kinect Energy Group
Elena Ehrlich - Data Scientist, Amazon Web Services

---

---

My personal notes:

• TODO: add notes

---

From the promotional materials:

M30-L: Investing in Technology Breakthroughs (Day 3 / 2:15pm - 2:45pm)

This session will outline managing investments in science and technology breakthroughs that challenge the status quo to bring futuristic ideas to life.

Josh Wolfe - Co-Founder and Managing Director, Lux Capital

---

My personal notes:

• "we're in a pretty good time right now for some of these moonshots"
• with respect to risk, "more things can happen than will"
• where to get edge = time arbitrage + science fiction
• there are about 25 VCs (venture capitalists) in attendance at this conference
• not as many people doing what we're doing, but there are a lot of folks competing looking at some of the same things
• it takes a behavioral disposition that's different, looking down the road
• where to focus: ghosts
• F. Scott Fitzgerald: "The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function."
• Mark Twain: "It ain't what you don’t know that gets you into trouble. It's what you know for sure that just ain’t so."
• Schopenhauer: "Talent is hitting a target no one else can hit."
• "best way to predict the future is to invent it"
• reading science fiction is often cited as the source of inspiration
• the gap between sci-fi and sci-fact is shrinking
• "the future is here, it's just unevenly distributed"
• Wolfe talked about how investments in one area lead to subsequent investments in related areas
• "that's the stupidest fucking idea I've ever heard": words Wolfe has said in reaction to something that is actually going to be a good investment
• what's important to think about is not when there is one thing, but when there is many, and the networking possibilities with them
• find the thing that nobody knows about, and try to solve the problems they're trying to solve
• dangerous words of investment: "this time it's different"
• what is the unpredictable impact when everyone has one?
• "Half-life of technology intimacy": you just know the way things are going to some extent: every half-life, the technology gets closer to you
• the directional arrow of progress is more and more pointing to you
• Wolfe explicitly invited emails from attendees

---

From the promotional materials:

A03-L: Follow the Money: What We Can Learn from Venture Capital Investment in Automation (Day 3 / 2:45pm - 3:15pm)

Venture capital firms are making big investments in the future of automation. Where they are putting their money is lending insight into the big trends that are expected to drive automation: mobility, artificial intelligence, drones, smart gripping, and machine vision — along with a few surprises.

Jeff Burnstein - Michigan, Association for Advancing Automation

---

My personal notes:

• Burnstein has 35+ years experience in robotics / automation
• robotics, machine vision etc initially largely failed in 1987 because of GM canceling a bunch of orders
• because of this, Burnstein began to wonder whether these were actually viable
• where the money is currently going:
  (1) mobility
  (2) gripping
  (3) AI
  (4) cloud
  (5) drones
  (6) vision
• mobility is giving machines legs
• a lot of money going into this
• only people have been able to do this type of task in the past
• a lot of gripper types out there, and not one gripper can grip everything
• everything is AI these days
• cloud robotics is applying the cloud to the robotics space
• Burnstein thought that the drones space would have declined by now, but he thinks he was wrong
• U.S. firms are now taking leadership with respect to drones
• industrial drones were originally a U.S. product, but there was no interest at the time, with union opposition, so it left the U.S. and went to Asia...
• now it's coming back
• one firm, Veo Robotics, is working to make robots aware of humans around them, as work is typically fully manual or fully robotic because robots are not aware of the environments around them
• Burnstein commented that robots have got to be about making our lives better, as it can't be limited to just competing against one another
• we know that when the number of robots increases, the number of jobs increases
• "robots are job creators"
• there are not enough people to fill jobs right now
• conference attendees have a lot of opportunities ahead

---

From the promotional materials:

M22: Edge AI Technology Is Redefining Smart Devices (Day 3 / 3:30pm - 4:30pm)

The ever-growing demands for privacy, short response time, and offline availability, create the trend of processing Deep Learning locally at the edge devices. Deep Learning demands high computation complexity and memory usage. Edge AI applications need the concurrent incorporation of Deep Learning, Image Signal Processing, 3D Graphics, and wireless connectivity, etc. Devices are resource-constrained, especially in memory bandwidth and thermal budget. These requirements and constrains make SoC design of AI-powered devices very challenging. In this session, we will cover the application and technology trends driving SOC solutions to make smart devices truly intelligent – now and in the future.

Ryan Chen - GM of Computing and AI Technology Group, MediaTek

---

My personal notes:

• face unlock takes about 0.2 sec locally and voice assistant takes about 0.5 to 1.5 sec via the cloud
• DNNs (deep neural networks) achieve good accuracy at the cost of high computation complexity and memory usage
• cloud based solutions are preferred in the early stages of DNNs for inference
• there is still network latency, even though time reduced from 2014 to 2016
• accuracy improved 24% between 2012 and 2016, but complexity went up
• complexity measured in terms of GMACs
• cloud vs edge
• more complicated AI will remain in the cloud
• cloud and edge will work together in a complimentary way
• there are design challenges for SoC: DNN workloads and device constraints
• the most important thing is to understand the entire system, not just the AI

---

From the promotional materials:

M09: Accelerating Machine Learning Projects (Day 3 / 4:45pm - 5:45pm)

Cut down on development time for machine-learning projects with third-party algorithms and models in a secure environment using Amazon SageMaker. In this session, we'll talk through the sample case of a machine-learning developer who wants to use machine learning to automate the auto insurance accident claim process.

Kanchan Waikar - Senior Solutions Architect, AWS Marketplace for Machine Learning, Amazon Web Services

---

My personal notes:

• TODO: add notes

---

From the promotional materials:

A08: A Faster Path to Smart Device Development with No Cloud Required (Day 4 / 9:00am - 10:00am)

The Alexa Connect Kit (ACK) provides a quick and easy path for device makers to connect devices to Alexa at a hardware level without building cloud services or complex networking and security firmware. Learn about ACK and the technical details of how one of the first devices to use ACK was prototyped in just a few weeks.

Rolando Cavazos - Division Vice President - NPD - Home and Personal Care, Spectrum Brands
Ben McInnis - Lead, Alexa Connect Kit, Amazon

---

My personal notes:

• TODO: add notes

---

From the promotional materials:

M41: Extracting Complex Concepts to Unlock Value from Clinical Language Data (Day 4 / 10:15am - 11:15am)

As healthcare shifts to the value-based world, providers, payers, and other stakeholders are seeking to improve patient outcomes and financials. Language data is abundant in the world of healthcare and offers a rich source of insights about patient stories, but extracting anything more complex than the simplest concepts requires highly manual and laborious effort. In this session, Roam Analytics will share how they use AWS to enable healthcare companies to extract complex concepts from clinical language data to understand patient population trends, build predictive models, and leverage more data to inform treatment choices.

Alex Turkeltaub - Co-Founder and CEO, Roam Analytics
Kyle Johnson - Practice Manager, Data Science, Amazon Web Services

---

---

My personal notes:

• patient data is largely in "language data" such as medical notes etc
• most of the structured data in EHR systems is what is billed
• macro trends exist in healthcare
• in the past, most of this data was "scribbled"...now most of this data is digitized, but is in "horrible shape"
• most of what you read in the papers about "outcome based" care is "baloney"
• what is changing is how payers are assuming risk
• there's a rise of paying for outcomes in a very different way
• "(IBM) Watson has done very poorly in the market"
• EHR systems have reduced the productivity of physicians because of their time spent typing
• things are going the way of recording patient / physician interactions
• most people talk about "concept extraction", i.e. does so and so suffer from such and such
• the point of language data is not who has heart failure, but what conversations have taken place
• "what Epic has done is criminal"
• Epic is purposely built to keep hospitals away from cheaper options
• and Epic is not interoperable
• models in healthcare will get worse over time because the way they do things changes over time
• models are created as follows:
  (1) take your data and annotate it
  (2) build predictive model on top of that data
  (3) deploy model
• there isn't a single hospital who can tell you who has heart failure!
• all they can do is tell you whether they've had heart failure in the past
• "physicians spend a lot of time doing really stupid things"
• with the Roam system, "bureaucratic" things get sent to operations, and medical things get sent to it
• "sounds prosaic, but is hugely complicated"
• 1/3 of the student body wanted to be physicians at Stanford when Turkeltaub was in school, and now this is down to 10%
• "a lot of drugs don't work in the real world like they do in clinical trials"
• paying attention to subtleties in the language data can point to why patients are not taking medication etc
• you get more predictive power from the unstructured data than from the structured data because of all the information that it provides
• it's important to understand that we have all this data
• "what the papers like to talk about is the sexy new drug...especially ones that fail", but these are largely irrelevant
• Turkeltaub presented a diagram listing the AWS services they are using, but commented that the analytics is done within the Roam platform
• the number one issue mentioned at a healthcare symposium he spoke at a month ago is that there is no standard workflow...
• software engineers have this, but data scientists and machine learning engineers don't
• don't forget about MLU (Machine Learning University)
• Turkeltaub mentioned during the Q&A that some have tried to do MDM (master data management) across different federal health agencies etc to come up with single definitions of things, e.g. because heart failure might be described in 14 different ways, but instead of making these 14 conform to 1 thing they just store all 14 and check against all of them
• it's hard to build a business case around small providers for various reasons, such as a lot of small providers making use of mom and pop shop EHR systems etc...
• so in order to penetrate this space there are a few options, such as providing access to Roam free of charge in exchange for some level of anonymized data in return so that Roam can feed this data into models on their end

---

From the promotional materials:

M42: Transforming Trash Collection: Turning Sensor Data into Machine Learning Models with Amazon SageMaker (Day 4 / 11:30am - 12:30pm)

Zolitron’s Z-Node autonomously captures data from the physical world like temperature, vibration frequencies, and GPS data. Zilitron uses Amazon SageMaker to create use-case specific machine learning models using these data streams. For example, Z-Node's vibration data is used to determine when a trash collection bin is or will be full, so that waste collection services can optimize their routes, saving time and energy. Zolitron will discuss how they clean data, and then build, train, and deploy machine learning models using AWS.

Arndt-Hendrik Zinn - Founder and CEO, Zolitron Technology
Shyam Srinivasan - Sr. Product Marketing Manager, AI/ML, AWS, Amazon Web Services

---

My personal notes:

• TODO: add notes

Amazon re:MARS 2019 Day 1 (June 4, 2019)

Amazon re:MARS 2019 
Aria Resort & Casino
Day 1 (June 4, 2019)

---

---

From the promotional materials:

KEY1: Opening remarks & Innovation Spotlights (5:30pm - 6:30pm)

Opening remarks by Dave Limp, SVP Amazon Devices and Services. Keynote to include innovation spotlights talks by (1) Marc Raibert, CEO, Boston Dynamics, (2) Morgan Pope, Associate Research Scientist, Walt Disney Imagineering, (3) Tony Dohi, Principal R&D Imagineer, Walt Disney Imagineering, and (4) special celebrity talk by Robert Downey Jr, Producer and Actor.

---

My personal notes:

• Dave Limp, SVP Amazon Devices and Services opened up
• there will be 3 innovation spotlight talks
• the original MARS (machine learning, automation, robotics, space) conference started about 4 years ago
• trying to get builders and dreamers together
• the gap that has existed between builders and dreamers is decreasing over time
• with respect to MARS, machine learning came first and is the foundation to everything 
• automation done correctly is invisible to the customer
• the goal of this conference is inspiration
• the party on Thursday will include rides around a track in pace cars, and battle bots
• Marc Raibert, CEO, Boston Dynamics presented the first spotlight talk
• gave interesting talk addressing the type of intelligence being addressed
• what kind of intelligence? the athlete vs the scholar
• Handle, Spot, and Atlas are 3 robots currently being worked on at Boston Dynamics
• Handle is purpose built, Spot is general purpose, and Atlas is R&D
• talked through some video playback of Handle moving boxes between pallets in laboratory warehouse setting
• then talked through video playback of Spot walking and climbing throughout the warehouse
• it appeared as though Spot was tethered whereas it seemed like Handle was freely moving about
• the first commercial product coming out in "a couple months" will be a version of Spot
• it sounded like the purpose of this product will be to monitor construction sites, apparently for compliance
• Marc then gave a demo of Spot physically on stage, and it was apparently let loose on the floor afterward (see my notes about this during the welcome reception below)
• the least time was spent discussing Atlas, he just made a few comments
• however, during this discussion he talked about the typical public perception of robots
• essentially, public perception is built around the concept of physical design combined with software
• however, Marc explained that the physical environment should be considered a third aspect
• video playback of Atlas moving boxes between tables was then shown, and Atlas also appeared to be tethered
• Morgan Pope, Associate Research Scientist, and Tony Dohi, Principal R&D Imagineer, both from Walt Disney Imagineering, jointly presented the second spotlight talk
• Morgan and Tony opened up by discussing the Abraham Lincoln automaton / animatronic from the 1964 New York World's Fair
• the original idea was to bring the 2-d world that Walt Disney was creating into the physical world
• I recall watching the Lincoln automaton several years ago at Disneyland, and as they were talking about it I thought perhaps it may have influenced the Philip K. Dick short story "We Can Build You", which I read when I was younger during my blitz of reading everything Philip K. Dick, but looking it up it appears it was originally written in 1962
• later, about 20 years ago, the desire was to untether such automatons from stationary positions like the 1964 Lincoln 
• they then talked through video playback of various incarnations evolving and being tested to do so, with the goal throughout this process to fail fast
• the original such mechanism resembled a raccoon, and they figured they needed to start making mechanisms in human form
• the next step was a human sized acrobat after they determined they could scale from earlier successes
• some level of decision making was then introduced to the acrobats in terms of when to drop etc
• when Tony was getting his PhD from Stanford, they would always say "that's really cool, but what's the business case?", but at Disney, "super cool is the business case"
• Robert Downey Jr then gave a monologue, and at this point I was no longer taking notes since I just wanted to sit back
• while his monologue was rather entertaining, Robert rambled quite a bit about dozens of topics for the duration of his time on stage
• the 2 most entertaining sequences were his periodic, presumably simulated interactions with Alexa, and his dialogue with Matt Damon
• at one point, Alexa abruptly interjected while Robert was talking, and he commented that he thought Alex was supposed to respond to inquiries, not lead
• it seemed as though Amazon may have been hinting that the role of Alexa may be changing down the road
• as much rambling as Robert may have done, I was pretty impressed with the extent to which he was able to memorize so many lines, including the back and forth with Alexa, and the back and forth with Matt Damon, which was via video that may have been prerecorded but seemed live
• the other thing I was impressed with was his ability to joke about his lack of education, and the years he wasn't doing so well as an actor due to personal challenges
• he also announced an initiative called "The Footprint Coalition" that he will officially launch in April 2020, the goal of which is to clean up the planet from an environment perspective, which can probably be done over the time span of only a decade according to some of whom with whom he has been meeting

---

---

From the promotional materials:

GEN1: Welcome Reception (6:30pm - 8:30pm)

We look forward to welcoming you to the event at the opening reception, which will take place in the Tech Showcase. This is a great opportunity to meet other attendees and familiarize yourself with what exhibits you may want to revisit later in the week.

---

My personal notes:

• not too many notes from me for this one, as I spent too much time watching the creations from Boston Dynamics walk about the exhibit hall
• I didn't realize until tonight that these are controlled by remote, at least for this conference, presumably for control
• it so happened by coincidence that a media interview with Marc Raibert started literally next to me
• as I was watching one of the other exhibits, one of the founders of acloud.guru happened to stop in front of me, and I immediately recognized him due to my working through one of his courses to pass my first AWS cert earlier this year

---

---

---

From the promotional materials:

W08: Practical Machine Learning, From Unlabeled Data To A Production Model (9:00am - 12:00pm)

Attendees will learn how to do end-to-end Machine Learning starting with an unlabeled dataset of images (I-Naturalist or similar), and use Amazon SageMaker to: explore and prepare the dataset, employ SageMaker Ground Truth to inexpensively label the dataset with object bounding boxes, train an object detection model (single shot multibox detector or SSD) based on the labeled dataset, deploy the model to an endpoint and evaluate the model performance, learn about hyperparameter optimization, and use HPO to find a more accurate model for deployment.

---

My personal notes:

• this workshop was led by Denis V. Batalov and Vikram Madan
• from what I understood, Denis leads ML as well as the specialist solution architects at AWS, and Vikram works on SageMaker Ground Truth
• Sagemaker is about 1.5 years old and was launched at re:Invent 2017  
• we largely followed the guide located at the following location: s3-us-west-2.amazonaws.com/sagemaker-remars/guide.zip
• the first workshop step was to set up labeling with Ground Truth
• prebuilt labeling workflow "bounding boxes" was used rather than other options such as image classification, text classification, semantic augmentation. or custom workflow
• Vikram commented at one point that "there always needs to be human input, since you're ground truthing it"
• needed to work in us-west-2 Oregon since that's where the dataset is located
• in following the instructions, I missed the step that suggested "providing some helper text to guide the workers for both good and bad examples"
• bounded boxes essentially are boxes that are drawn around the object of interest in each image, and for this workshop, the object of interest was bees
• good bounding box examples include all visible parts of the insect: legs, antennae etc, and bad bounding box examples are when the bounding box is too big and/or excludes some visible parts of the insects
• I talked to one of the AWS assistants at one point to ask about my not noticing the note in the Jupyter notebook to provide this helper text for workers, as it seemed to be taking a while for them to complete the tasks I had set up, but I wanted to see what would happen
• the sample size of 500 images we used here was 2%, resulting in 10 images for which workers needed to created bounded boxes
• he told me that 30-45 minutes should be expected for the 10 image sample, and his response to my comment that this seemed to be a long time was that you can pay more to get the work done more quickly
• I'm not sure he followed everything I was saying, as I asked him how accountable the workers are in terms of accuracy of their work: are random samples perhaps taken and assessed to determine label accuracy? I asked twice, and he just kept saying that they are under contract with Amazon
• Vikram commented about the crowdsourced global workforce available via the AWS marketplace: since it's an art, the price should be set higher if the task is more difficult / takes more time to execute
• Vikram commented at some point that "I'm not a Mac person, can someone give me a hand?", and our table of developers was joking about this, as all of this work was just being performed through a browser anyway
• Vikram also said that "there's variability in terms of how long it will take the task to complete"
• Denis later said that "hyperparameters are the interesting bit": the weights, essentially metaconfiguration
• my labeling job never finished, and got stuck on 9 out of 10 images complete
• it's possible that my job would eventually complete, but most workshop attendees were waiting for workers to complete this task, and I needed to start cleaning up AWS resources to prevent further charges (we were all issued $50 credit cards as reimbursement for charges we incurred during the workshop)
• the demo by Denis didn't finish either
• Denis asked if anyone was having any issues, and a couple attendees said that only a certain instance type was supported, to which Denis responded that a limit increase request would need to be submitted to AWS, which was interesting because if this were the case we should have received a limit increase prior to the workshop
• I was fortunate to have a woman at my table, as I did some quick math that considered about 21 tables with 8 people each, totaling 168, rounding down to 150 since some chairs were empty, and seeing only about 5 women total in attendance
• unfortunately, very little here that I could have just learned by following some AWS tutorials, and the pace of this workshop was extremely slow, with the presenters providing very little value add
• in addition, I was disappointed that this workshop did not end up being the full day that was originally advertised when I registered for the conference, as conference organizers apparently ended up deciding to instead offer 2 sessions, each only a half a day in length, after they realized how popular it was to attendees
• as with other conferences, however, the great aspect was meeting people at my table who traveled from Denver, Atlanta, Dallas, and Tokyo to attend the conference, and I talked at length with the gentleman from Denver who is a university professor there who teaches statistics and also does consulting

Implementing Databricks for Data Science and Engineering Teams: Part 1

---

A case study I wrote about a recent client project implementing Azure Databricks for data science and engineering teams was recently published (April 9, 2019). This write-up is the first of a two part series covering the first half of a client engagement with my team.

(The original post can be found here, link expected to change, and my LinkedIn announcement of this post here.)

---

The analytics arm of a large midwestern U.S. insurance firm needed to analyze its current state architecture used by its teams of data engineers and data scientists. The company asked SPR to help by focusing on the portion of its architecture making use of Azure HDInsight, and building a proof of concept (PoC) on Azure Databricks Unified Analytics Platform for the company’s future state architecture.

The company had 3 key criteria for implementation of the PoC:

• Focus on use of the R language for machine learning (ML) models
• Account for current data science team development processes
• Preference for use of Azure Data Factory for data pipelines, which the current data engineering team was already looking to adopt

Using these criteria, SPR first worked with the data science team to understand current development processes and to define the PoC scope.

Background on HDInsight and Databricks

At the time, the data scientist team used Azure HDInsight as its primary tool for executing R language ML models with Apache Spark. Of note, Azure HDInsight is an Azure distribution of Hadoop components from Hortonworks Data Platform (HDP), which has been one of the three most recognized Hadoop distributions alongside Cloudera Distributed Hadoop (CDH) and MapR Data Platform (MDP).

Apache Spark is a separate component, which provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. Since the data science team primarily made use of Azure HDInsight for executing ML models on Apache Spark through Apache Hadoop YARN, the fact that Hadoop was being used to do so resulted in the current state ecosystem extending considerably beyond Apache Spark.

Another product, Databricks, was made generally available for Azure in March 2018, just 6 months prior to the inception of the SPR engagement (Databricks was made available for AWS in June 2015). Azure Databricks is an Azure cloud optimized analytics platform based on Apache Spark. It provides fully managed Spark clusters, an interactive workspace for exploration and visualization, and a platform for powering Spark-based applications.

The Azure implementation of Databricks is so tightly integrated with Azure that the service is a first-party Microsoft offering provisioned as with other Azure-branded services such as Azure HDInsight, even while the Databricks product itself is from a third-party (albeit designed in collaboration with Microsoft).

While Azure HDInsight as a product offering is not expected to go away any time soon, Microsoft claims its Azure Databricks offering provides the industry’s fastest Spark implementation: 5 times faster than Apache Spark (which its creators already claim runs workloads up to 100 times faster than Apache Hadoop).

Databricks also comes packaged with a built-in Hive Metastore providing backward compatibility via Spark SQL with a common abstraction layer for data currently being used with HDInsight. It was created by the founders of Spark who continue to innovate, with recent RStudio collaborations that bring an option to integrate RStudio Server with Databricks, as well as R support for its open source MLflow product, as examples.

Since it was the latest version available when the project started, SPR largely focused on Azure Databricks 4.3, which uses Apache Spark 2.3.1. Databricks 5.0, which uses Apache 2.4.0, was released just a week after the new Spark version was released in November 2018, which makes sense knowing that Databricks comes from the creators of Spark.

Contrast this situation with Azure HDInsight 4.0, which is in preview mode and still makes use of Spark 2.3.1, first released in early June 2018. While just a few months apart from one another, remember that Databricks 5.0 together with Spark 2.4.0 resolves over 1,000 outstanding tickets (not to mention availability of new SparkR features applicable to this data science team), so earlier release dates are not just a matter of semantics: there are real benefits to getting access to releases sooner.

Assessing Available Tooling

With things moving so quickly with Databricks and the broader Azure ecosystem, SPR assessed what was available – both for products ready for production, as well as those still in the works and so not yet mature enough for adoption. And while it is important to discuss what tooling we used, it is also important to understand what we didn’t choose and why. Let’s take a look at some examples:

Azure Machine Learning service – SPR determined Azure Machine Learning service, the third incarnation of a Microsoft offering, was not ready for use (it was not made available until the final week of SPR’s final deliverables).

MLflow – We also chose not to use an open source framework from Databricks called MLflow, which had just added R language support the week prior to the project start. As the data science team had just migrated away from SAS, it was especially important to assess the level of available R support for needed Databricks features, at least for an initial migration. That said, we recommended that the company keep a close watch of this framework, as its purpose is to close the gap that currently exists in standardizing the end-to-end ML life cycle.

Other tooling we did not use included additionally available Microsoft tooling, because the data engineering and data science teams already make use of Azure:

Microsoft Visual Studio 2015 and 2017 – We assessed Microsoft Visual Studio 2015 and 2017 specifically with respect to R and Azure related tooling. While we noticed availability of a “Microsoft Azure Data Factory Tools for Visual Studio” extension for Visual Studio 2015, we also saw that this extension is not available for Visual Studio 2017, and that there are apparently no plans to incorporate comparable tooling in Visual Studio 2019 or later releases of Visual Studio 2017 (although the Visual Studio roadmap hints at some possibly relevant features for 1Q2019).

Microsoft R Tools for Visual Studio, Microsoft R Client, Microsoft Machine Learning Server – All of these products are interrelated because Microsoft R Client is used to remotely connect to Microsoft Machine Learning Server, and Microsoft R Client is available for this connection via Visual Studio 2017. We successfully demonstrated the setting up of a remote connection from Visual Studio 2017 to a Microsoft Machine Learning Server instance in Azure after working through largely undocumented configuration setup. However, we did not recommend using this tooling because it does not seem to be positioned well.

Databricks Proof of Concept (PoC)

Once we determined what we would not be using for the PoC, we moved forward with the other Azure components that we would be using. In addition to Azure Databricks, we chose Azure Blob Storage, Azure Data Factory, and Azure DevOps alongside desktop components such as Databricks CLI, PowerShell, RStudio Desktop, and Git.

We also installed RStudio Server to the driver node of the Databricks cluster. Each of these components plays a part in two sets of workflows we worked through for the data engineering and data science teams. This helped them understand real-world application of the tooling for their needs, but also helped demonstrate to them what actually works (as some available documentation is either incomplete or inaccurate) and how to perform the work in a recommended manner.

The first workflow we shared was one we deemed “DevOps.” It comprised 6 key tasks:

1. creating and terminating a Databricks cluster
2. using a Databricks cluster initialization (“init”) script
3. installing RStudio Server on a Databricks cluster
4. integrating with Azure DevOps
5. creating and installing an R package for a model
6. testing and measuring performance

Custom scripts were developed in a manner which enables repeatable and consistent execution of DevOps tasks. Databricks support for RStudio Server integration as of June 2018 was a highlight, as it brings a similar look-and-feel and functionality as the RStudio Desktop product used by the data science team. Plus, it includes the ability to execute code apart from Databricks notebooks, something the data science team requested.

Additionally, the data engineering and data science teams wanted to integrate directly with source control. We demonstrated several options for this purpose, including our own recommendation to use Azure DevOps, which the data engineering and data science teams have since adopted.

We called the second workflow we shared “data pipeline”. It also had 6 key tasks:

1. creating a mount point to Azure Blob Storage
2. accessing, reading, and preparing input data
3. writing output data
4. manually executing notebooks and models
5. using Azure Data Factory to automate execution
6. testing and measuring performance

As with DevOps tasks, data pipeline tasks were scripted as much as possible. The data engineering team was already moving in the direction of Azure Data Factory, so we used this product in the PoC to execute code in a batch manner. And as Azure Blob Storage is currently being used by the data science team with HDInsight, using Azure Blob Storage here with Apache Hive compatible Spark SQL showed how to migrate from HDInsight to Databricks in a future project phase.

The Result

After we presented our findings and delivered the scripts we developed, the firm decided to move forward with a recommended Pilot phase. The purpose of the Pilot phase is to migrate to Databricks a subset of ML models currently being executed by the firm in production using HDInsight. As part of this migration, testing and measuring performance are also planned.

Technologies used during this effort included Azure Databricks, Azure Blob Storage, Azure Data Factory, RStudio Server, Azure DevOps, R, Python, Databricks CLI, PowerShell, RStudio Desktop, and Git.

Manuscript Review: "Project Decisions"

An editor for a publisher reached out to me via LinkedIn on January 8, 2019 with an opportunity to provide input on the second edition of a book in exchange for a flat fee. I was given 1 calendar month to complete the review, and I submitted my feedback on February 9, 2019.

While I've reviewed quite a few technology and business texts over the years, including those at the request of authors and publicists, this was the first time I was paid to perform a review.

In addition to providing me the book manuscript in Microsoft Word format, the editor also posed a number of questions that she wanted me to address, so my deliverables to the editor were two-fold: (1) an edited book manuscript with my inline comments, and (2) a 4-page document that I wrote to address her questions. My work on this manuscript review, including a reading of the book, was performed in 1 calendar day.

At some point I'd like to write a post about all of the reviews I've performed directly for authors and publicists. Some colleagues have asked why I do this for (typically) no pay, just like I've been asked why I do pro bono work.

While my reasons for doing pro bono work differ, the following are my reasons in a nutshell why I perform these reviews: (1) to grow my network, including for a time when I might want to be published myself, (2) to develop my writing and editing skills, and (3) because I find it enjoyable to give back from my work experience.

The following is the text from my manuscript review, including the bolded editor questions.

---

Review by Erik Gfesser for Charlotte Ashlock and Sarah Modlin at Berrett-Koehler Publishers “Project Decisions: The Art and Science (Second Edition)”, by Lev Virine and Michael Trumper:

If possible, it would be great to also receive a digital copy of the manuscript with your inserted notes and observations. I will share your content review with the author and the editor on the project.

In addition to this document, I am also providing an edited manuscript with review notes constrained to the material added for the new edition, which is understood to be highlighted in yellow. My review of this new content could not help but make note of a considerable number of grammar and spelling mistakes. As a prolific reader and writer, I often get annoyed when coming across such mistakes in published material because my presumption is that the authors of such works should be better writers than myself. And as a reviewer, I tend to rate books lower simply because of such mistakes. Most books have a small amount of grammar and spelling issues, but it is extremely distracting to me as a reader when I need to additionally think through what I am reading simply because of the manner in which the material was written. As the manuscript provided to me is likely not the first draft, these issues should have been resolved by now, and I highly recommend that the second edition not reach publication until the notes I have added to the manuscript have been addressed. Note that my inline review notes were written during my initial walkthrough of the manuscript, so this document includes additional thoughts following this additional reading (apart from my notes on grammar and spelling issues, which are constrained to the manuscript).

What additions to the new edition are you the most pleased about? What do you think is most compelling about the new and revised version?

Page 23, 197-198

The evolution of Tesla Model 3 manufacturing over such a short time period is bound to be a classic example for a long time to come, regardless of whether this car model has long-term success. As an avid Tesla follower myself, I think it will, and as such I think this addition to the new edition is a good one and will remind the reader that they are reading a recently published book, a book attribute that some will undoubtedly find of value.

What are parts of the book you think are working less well? Please give examples of where you see this, and how you would revise it to make it work better.

Page 12:

The “Test Your Judgment” section includes a question which makes use of an example citing Kim and Kanye, and the text for the new edition qualifies these individuals as being Kim Kardashian and Kanye West. I recommend use of a different wedding couple, although I realize the intention is probably to make use of a well-known couple while still being as apolitical as possible. One of the reasons for this suggestion is to help prevent this book from becoming outdated when this couple's popularity fades away. I suggest use of a royal couple instead. What I'm not immediately clear about here is that if this phrase is what was added from the first edition, how anyone would have known when reading the first edition who "Kim and Kanye" might be in the remainder of the example.

Page 13:

Similarly on the following page, my presumption is that “Mike Zukkerfield” is a play on "Mike Zuckerberg", in which case I'm not sure why this name wasn't used to begin with, especially since "Kim and Kanye West" were already cited earlier. Why wasn't something like "Kim and Kanye East" not chosen if there is a concern about citing real individuals? Alternatively, I would choose another name here since Mike Zuckerberg is losing favor as of late with the public.

Page 39:

Inclusion of Figure 2.4 is understandable, but I recommend a photo that shows the car from a different enough angle so that the logo is not visible. The reality is that I naturally looked at the image before reading the sentences leading up to it, and already knew that the depicted car is not from Mazda, Toyota, or Kia simply by my not recognizing the logo.

Page 49:

I am not sure whether Figure 3.1 provides much value to the discussion other than possibly to help the reader visualize the relationships between the 3 components. If it is decided that this image does provide value, I recommend replacing it with one that more properly fits the word "Comprehensiveness", or using something like "Compre-" and "hensiveness" so that syllables are broken down correctly.

Page 63:

While the author should be careful in general when using drug firms as an example here, the author’s added remark for the second edition that “a significant number of drugs never make it through the approval process” should be used with caution. While this statement might be technically true, just keep in mind that many drugs are fast-tracked, the ramification being that some risks might not be discovered until a later point. In addition, my understanding is that drug firms have enjoyed zero liability for vaccine injury as of the 1980s. The number of injuries was getting so high at the time that firms threatened to stop making vaccines unless they were granted immunity.

Page 135:

It might be worth noting that not only can everything be benchmarked, but everything can be measured. The practice of measuring everything is an important one because I have often heard from others in the workplace that many aspects are not measurable. The reality is that everything is measurable, but it needs to be determined whether doing so is worth the cost, financial or otherwise.

Page 147:

I have extensive experience with agile processes (and have multiple Scrum certifications), and Figure 12.5 of the waterfall model is not accurate. Instead, this diagram represents how individuals have misinterpreted the model. While I am definitely not an advocate of traditional waterfall methods, keep in mind that each phase as originally defined was supposed to loop back to its prior phase to act like a feedback loop. The author who first discussed this model included these feedback loops, but others have since excluded. If you wish to include this diagram in the second book edition, I recommend either calling this out, or explain in the context of how the waterfall method is typically carried out.

Pages 201-204:

The “Event Chain Diagrams” section needs to be cleaned up, and I have a few recommendations with consideration for its inclusion within a broader “Basic Principles of Event Chain Methodology” section that lists six principles. What is confusing to me as a reader is why diagrams are listed as a principle, and if these are really a principle why section principle #3 is not explained prior to the diagrams of principle #1 and principle #2. I think it makes sense to explain diagram notation before the diagrams of these earlier principles, and as such I recommend either reconsideration of whether event chain diagrams are a principle or just a communication mechanism used for the methodology. Either way, perhaps an introductory sidebar should instead first explain how to read such diagrams. Additionally, the author should be consistent in their structure of each bullet point. For example, I mention in my manuscript notes that each bullet should follow the pattern of symbol followed by explanation. The author goes back and forth with this pattern and explanation followed by symbol, and I found myself restructuring each bullet point as I was reading through all of these, distracting myself from likely author intent.

Page 252:

The author’s list of technologies to solve problems with scarce resources should probably be framed within the context of the fact that these technologies also pose risks and costs. For example, as someone doing "big data" work I know that while this type of work can provide significant value to an organization, this is not always the case, with "small data" sometimes providing a more relevant solution. Additionally, the terms "big" and "small" in this context have been largely irrelevant for approximately the last 5 years and I don't make a habit of using them.

Pages 254-256:

Appendix A provides a table of risk and decision analysis software. I highly recommend that the paragraph preceding this table be elaborated to explain that every customer of such software will have their own unique requirements, so whether these will be of help is only a possibility. If the author has not already done so, I also recommend that research reports from firms such as Gartner and Forrester be consulted as input to this table to help ensure that a survey of software within the domains of quantitative project risk analysis software, enterprise risk management software, and “other” decision and risk analysis software used in project management covers enough ground, and that high-level considerations for choosing a software vendor be listed. Additionally, I would define the difference between the categories of the first two tables, because it’s not clear to me as a reader what distinguishes these two. The author should also consider breaking down these tables further or adding a column that indicates subcategory of software. Alternatively, every single entry in the first table is described as Monte Carlo software: if this type of software really dominates “quantitative project risk analysis software”, why not just describe this table as “Monte Carlo software” instead?

What themes and topics are missing from the new edition that you would really like to see added into the new edition? Where would you like the author to go into greater detail? Is there something the author does not discuss that you wish she had?

In reviewing what the author presents about “agile”, I think it’s important to note that the concept of agile is not a monolith. Additionally, not everyone views agile methodologies as forms of project management, with many going so far as to say that project management in many cases is not even needed, and I share this opinion. I strongly recommend going into much greater detail on this subject matter, since agile methodologies have had such a profound impact on how work is performed. For example, I recommend breaking down the agile universe a bit to discuss some of the key approaches. While agile methodologies were still relatively uncommon in 2007 when this book was first published, the ensuing 10 years have seen widespread adoption, for better or worse. As such, the author should include discussion as to the scenarios in which each of the key approaches is more appropriate, especially within the context of corporate culture, which I’ve found to be a weighty success factor.

Are there any parts of the revised manuscript that you would recommend cutting or shortening and why? What do you think is the least crucial information within the book, if cuts have to be made?

I would remove chapter 5 (“Creativity in Project Management”). Additionally, I would not add Appendix A (“Risk and Decision Analysis Software”) unless the author addresses my comments earlier in this document.

How does the new edition deepen and strengthen the themes and contributions found in the original work? What do you personally find most interesting in the work and most useful for your own work and life? What do you like least?

My own work has increasingly concentrated in three areas: enterprise architecture, data and database architecture, and custom software development. Since reading the first edition of this book, I have increasingly come to the conclusion that in all of these areas (which cover a significantly large portion of corporate technology work), I largely see no need for project management. As such, I personally do not find much practical value here for my own work. That said, I am always open to new ideas, and reading books such as this one provides different perspectives that will inevitably feed my evaluation of work approaches for clients in the workplace. And of course, not everyone shares my opinions, especially those working in firms who approach work in a traditional manner.

Personal Reading List: 2Q2019

My personal reading list for April, May, and June 2019.

Color Key: Special Notes, Completed Reading. Updated throughout the Quarter.

---

At Bat (Schnell)

---

The Model Thinker: What You Need to Know to Make Data Work for You, by Scott E. Page, Basic Books, 2018.

Thinking Architecturally: Lead Technical Change Within Your Engineering Team, by Nathaniel Schutta, O'Reilly, 2018. Copy provided by Pivotal.

---

 At Bat (Langsam)

---

Kafka: The Definitive Guide, by Neha Narkhede, Gwen Shapira, and Todd Palino, O'Reilly, 2017. Copy provided by Confluent.

Practical Statistics for Data Scientists: 50 Essential Concepts, by Peter Bruce and Andrew Bruce, O'Reilly, 2017. Copy provided by MarkLogic.

---

 On Deck

---

Apache NiFi for Dummies, by Christopher Gambino, Dan Rice, Joseph Niemiec, Mark Johnson, and Jordan Martz, Wiley, 2018. Copy provided by Hortonworks and Attunity.

AWS Certified Solutions Architect Associate All-in-One Exam Guide (Exam SAA-C01), by Joyjeet Banerjee, McGraw-Hill Education, 2018.

---

In the Hole

---

Designing Autonomous Teams and Services: Deliver Continuous Business Value through Organizational Alignment, by Nick Tune and Scott Millett, O'Reilly, 2017. Copy provided by O'Reilly.

Introducing Istio Service Mesh for Microservices, by Christian Posta and Burr Sutter, O'Reilly, 2018. Copy provided by Red Hat.

New Book Review: "Applied Predictive Analytics"

New book review for Applied Predictive Analytics: Principles and Techniques for the Professional Data Analyst, by Dean Abbott, Wiley, 2014, reposted here:

Abbott is sure to discuss his philosophy and approach at the outset. "This book describes the predictive modeling process from the perspective of a practitioner rather than a theoretician. Predictive analytics is both science and art. The science is relatively easy to describe, but to do the subject justice requires considerable knowledge of mathematics. I don't believe a good practitioner needs to understand the mathematics of the algorithms to be able to apply them successfully, any more than a graphic designer needs to understand the mathematics of image correction algorithms to apply sharpening filters well. However, the better practitioners understand the effects of changing modeling parameters, the implications of the assumptions of algorithms in predictions, and the limitations of the algorithms, the more successful they will be, especially in the most challenging modeling projects."

"Science is covered in this book, but not in the same depth you will find in academic treatments of the subject. Even though you won't find sections describing how to decompose matrices while building linear regression models, this book does not treat algorithms as a black boxes. The book describes what I would be telling you if you were looking over my shoulder while I was solving a problem, especially when surprises occur in the data. How can algorithms fool us into thinking their performance is excellent when they are actually brittle? Why did I bin a variable rather than just transform it numerically? Why did I use logistic regression instead of a neural network, or vice versa? Why did I build a linear regression model to predict a binary outcome? These are the kinds of questions, the art of predictive modeling, that this book addresses directly and indirectly."

Amazon provided me a copy of this book to review after it was first published, and since I have a personal policy never to write a book review until after it has actually been read, this review is unfortunately being written a bit late. It has been more than 4 years since this book was published, and because the field being discussed has evolved considerably during this time period, it is showing its age at least a little bit, but this is mainly contained to Chapter 12 ("Model Deployment"). Considerable strides have been made in the namesake task being considered in this chapter, model deployment, but much can be said for artifact deployment in non-analytics areas of work as well. However, it is in this chapter where Abbott also covers various deployment options, and one of my only grievances about the content the author has to share is his assertation that deployment of models to databases is actually a viable option. While I realize that some commercial products now provide this capability, modelers should follow the best practices of software development not to embed logic in databases.

After the author provides an overview of predictive analytics, he delves into setting up the problem, understanding data, and preparing data, followed by discussions on itemsets and association rules, descriptive modeling, interpreting descriptive models, predictive modeling, assessing predictive models, model ensembles, text mining, model deployment, and case studies. Potential readers of this book might be interested in knowing that the author references a process called CRISP-DM (Cross-Industry Standard Process Model for Data Modeling) throughout, mainly because custom processes typically fall in line with the steps outlined by this process, and it helps describe the most commonly applied steps in the process, arguably providing needed structure for practitioners that reminds them of steps that need to be accomplished including the documentation and reporting especially valuable for new modelers.

The chapters that I especially appreciated are chapter 3 ("Data Understanding"), chapter 4 ("Data Preparation") and chapter 5 ("Itemsets and Association Rules"), which in my view cover topics often skipped or downplayed by other texts. As the author comments in the summary to chapter 3, the processes involved in understanding the data after it has been collected cannot be rushed: "problems missed during data understanding will come back to haunt the analyst during modeling". Chapter 4 also provides warnings. "Do not consider data preparation a process that concludes after the first pass. This stage is often revisited once problems or deficiencies are discovered while building models." Additionally, "Overfitting the data is perhaps the biggest reason predictive models fail when deployed. You should always take care to construct the sampling strategy well so that overfitting can be identified and models adjusted appropriately."

The bulk of the material, however, is concentrated in chapters 6 through 11, and while the material on descriptive modeling, prescriptive modeling, model ensembles, and text mining is presented well, these discussions largely provide a survey of the different approaches, and in 2019 more details can be found elsewhere. One aspect that potential readers should keep in mind is that while the author focuses on the practical, this book doesn't provide any code and doesn't discuss any open source frameworks or commercial products. In a sense the author rises above such specific implementations, so come here for practical approaches and lessons learned, but not hard details beyond the functional. However, the functional is often what data professionals need, as I have often found from my own consulting experience that field practitioners often can't see the functional forest for the framework trees.

Media Query Source: Part 17

The responses I provided to an "anonymous" media outlet series of queries on February 27, 2019:

Media: Facebook's chief AI scientist recently said that a more flexible and easier programming language, versus Python, for deep learning is needed. Yann LeCun also discussed the need for new hardware to support self-learning for deep learning systems. Would a more accessible programming language, or one that simply adds another option for deep learning development, help to bring more potential AI/ML staffers into the fold? The idea could hold some appeal for firms looking to jump into the AI pool but finding the staffing levels shallow. If such an idea does make sense, how can organizations be convinced to move away from Python?

Gfesser: The opening question here is about staffing shortages specific to machine learning, but what LeCun had to share seems mainly focused on the technology.

 

LeCun commented that "there are several projects...to kind of design such a compiled language that can be efficient for deep learning, but it’s not clear at all that the community will follow, because people just want to use Python."

 

The reality is that hundreds of programming languages exist, but only a relatively small subset are commonly used in industry. And when these commonly used languages are filtered for use on data analytics projects, only a handful remain.

 

And languages which find favor (and find favor for specific uses) within the development community change over time. While Python has been around for decades, it has only gained widespread popularity in just the last few years.

 

In prior years before Python gained favor, proprietary commercial packages reigned alongside the R language for quite some time. When I chose a language to use for a data analytics project in 2012, I chose R, but just within the space of a couple years Python started gaining a foothold for this type of work. I am using R alongside Python on a client project right now, but the bulk of the new code we are writing is in Python.

 

Notice that I'm not even talking about technology benefits and drawbacks here: I'm talking about popularity. Yes, there are technical reasons why languages become popular, but in the case of Python much of the popularity is due to its status as a general purpose language, not a language built specifically for statistics like R, which tends to be favored by academics and not industry professionals. And in the early years of the 2010s, open source frameworks such as Pandas started to be developed by the community which provide much of the same functionality as languages such as R which are not as intuitive to use.

 

When LeCun mentions the possible need for "compiled languages", it is a bit distracting because interpreted versus compiled is a broader discussion. I think the intent here was to address relative performance of programming languages, in which case he probably has a point. But the history of programming languages shows us that the "best" languages often do not win, and "best" is a relative term that depends on the goals of those using a given programming language. Often what is right for some is not right for others. 

 

The GitHub Octoverse report cited by this article mentions that Python is the most commonly used language used in GitHub repositories tagged "machine learning", but look at the other languages listed for this tag: a relatively new language used for analytics called Julia is #6, and all languages above it in the list are general purpose languages. Will Julia ever take hold of an increasingly large market share of languages used for machine learning? My bet is against it, as Python beat it to the punch. Is it more elegant? Is it more performant? Perhaps, but developers generally don't want to learn multiple languages to get the job done: this is where Python enters the picture.

 

Sure, there is room for more languages, but there will need to be viable reasons to broadly adopt, and these will likely need to fit within the realm of general purpose rather than specialty. The exception being that if automated ML begins to take hold on a large scale, and these new languages are used to build the services providing the automated ML, developers will not necessarily need to learn these new languages since they will be constrained to black boxes that developers will just need to call with general purpose languages such as Python, which in the view of many developers is already an accessible programming language.

---

Media Query Source: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9, Part 10, Part 11, Part 12, Part 13, Part 14, Part 15, Part 16

New Book Review: "An Introduction to Machine Learning Interpretability"

New book review for An Introduction to Machine Learning Interpretability, by Patrick Hall and Navdeep Gill, O'Reilly, 2018, reposted here:

The content of this freely made book provides an introduction to the perceived need for machine learning interpretability, and how satisfaction might be fulfilled by practical methods. As the authors explain in their introduction, trusting models and their results is a hallmark of good science. "The inherent trade-off between accuracy and interpretability in predictive modeling can be a particularly vexing catch-22 for analysts and data scientists working in regulated industries. Due to strenuous regulatory and documentation requirements, data science professionals in the regulated verticals of banking, insurance, healthcare, and other industries often feel locked into using traditional, linear modeling techniques to create their predictive models. So, how can you use machine learning to improve the accuracy of your predictive models and increase the value they provide to your organization while still retaining some degree of interpretability?"

"This report provides some answers to this question by introducing interpretable machine learning techniques, algorithms, and models. It discusses predictive modeling and machine learning from an applied perspective and puts forward social and commercial motivations for interpretability, fairness, accountability, and transparency in machine learning. It defines interpretability, examines some of the major theoretical difficulties in the burgeoning field, and provides a taxonomy for classifying and describing interpretable machine learning techniques. We then discuss many credible and practical machine learning interpretability techniques, consider testing of these interpretability techniques themselves, and, finally, we present a set of open source code examples for interpretability techniques."

This terse 40-page work is broken down into the following sections: "Machine Learning and Predictive Modeling in Practice", "Social and Commercial Motivations for Machine Learning Interpretability", "The Multiplicity of Good Models and Local Locality", "Accurate Models with Approximate Explanations", "Defining Interpretability", "A Machine Learning Interpretability Taxonomy for Applied Practitioners", "Common Interpretability Techniques", "Testing Interpretability", and "Machine Learning Interpretability in Action". Additionally, links to the 15 papers that fed the content of this book are provided in the concluding "References" section, such as "The Evolution of Analytics: Opportunities and Challenges for Machine Learning in Business", "Why should I trust you?: Explaining the predictions of any classifier", "Towards a rigorous science of interpretable machine learning", "The Promise and Peril of Human Evaluation for Model Interpretability", and "Ideas on interpreting machine learning".

The bulk of the content rests in two sections: "A Machine Learning Interpretability Taxonomy for Applied Practitioners" and "Common Interpretability Techniques". This first section provides a summary of the previously defined taxonomy in the last paper listed above. As the authors explain, the complexity of a machine learning model is directly related to its interpretability: "Generally, the more complex the model, the more difficult it is to interpret and explain. The number of weights or rules in a model—or its Vapnik–Chervonenkis dimension, a more formal measure—are good ways to quantify a model’s complexity. However, analyzing the functional form of a model is particularly useful for commercial applications such as credit scoring." Such functional forms can be broken down into various degrees of interpretability: (1) linear, monotonic functions provide high interpretability, (2) nonlinear, monotonic functions provide medium interpretability, and (3) nonlinear, nonmonotonic functions provide low interpretability.

This second section discusses what the authors describe as credible techniques for training interpretable models and gaining insights into model behavior and mechanisms, in the context of the aforementioned taxonomy. While many of these techniques have existed for years, many others have only been put forward relatively recently in a flurry of research: "The section begins by discussing data visualization approaches because having a strong understanding of a dataset is a first step toward validating, explaining, and trusting models. We then present white-box modeling techniques, or models with directly transparent inner workings, followed by techniques that can generate explanations for the most complex types of predictive models such as model visualizations, reason codes, and variable importance measures. We conclude the section by discussing approaches for testing machine learning models for fairness, stability, and trustworthiness."

While this book is relatively short, this section provides dozens of links across the "References" and "OSS" (open source software) portions of the discussed techniques, including links to two additionally freely available books, "The Elements of Statistical Learning (Second Edition)" and "Introduction to Data Mining (Second Edition)", as well as additional papers interestingly not included in the aforementioned "References" section at the end of this book, such as "Interpreting Blackbox Models via Model Extraction", "Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Explanation", and "A Unified Approach to Interpreting Model Predictions".

Some of these papers might not be seen as practical by some potential readers, who will instead be interested in the links provided for several GitHub repositories, including one specifically associated with this book, although I was disappointed to find that at least two of these repositories are associated with Oracle or IBM. However, apart from these repositories are links to several Jupyter notebooks that some will likely find of value, such as "Engineering Transparency into Your Machine Learning Model with Python and XGBoost", "Increase Transparency and Accountability in Your Machine Learning Project with Python and H2O", "Testing machine learning models for accuracy, trustworthiness, and stability with Python and H2O", and "Explain Your Predictive Models to Business Stakeholders using LIME with Python and H2O". Unfortunately, I discovered that one link provided by the authors points directly to a tarball file, but a decent list of follow-up reads nonetheless.