Erik on Software

Media Query Source: Part 45 Reworked (US digital magazine)OpenAI & the Databricks-Microsoft AI allianceDatabricks available on AWS, Azure, and GCPMicrosoft providing AI options makes sense

---

My responses ended up being included in an article at Reworked (September 7, 2023). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The responses I provided to a media outlet on August 29, 2023:

Media: What does Databricks do and what can it bring to Microsoft Azure?

Media: Recently, Microsoft announced that it is looking at offering Databricks AI through Azure. This would allow Microsoft to offer Databricks' machine learning (ML) and data analytics tools to Azure customers as part of its cloud services.

Databricks provides an AI-powered data analytics platform to help organizations build their own AI models.

However, this deal contrasts with OpenAI's approach of developing proprietary AI models and licensing them to partners like Microsoft. The two companies work closely on Microsoft 365, Windows, Bing and more.

By offering Databricks' software on Azure, Microsoft aims to meet the growing enterprise demand for customized AI tools. This allows companies to leverage AI in applications tailored to their specific business needs.

Gfesser: I first made use of Databricks in 2018, since the time it was made generally available (GA) in Azure.

My first experience while with SPR Consulting involved addressing all of an insurance client's issues in their unsuccessful attempts to adopt Databricks. This work included building a slew of proofs-of-concept (PoCs) demonstrating how to do everything they were looking to do, followed by migrating their machine learning (ML) models to Databricks to run a pilot, as well as providing recommendations on how team processes were being executed and how data pipelines were designed.

While with Deloitte these past two years, I've driven several initiatives that started with the roadmap I laid out for phased implementation of a data lakehouse with Databricks. While Databricks was already being used in production, few knew this because it had been solely implemented as a back-end engine to execute the workloads sent to it by a non-Azure third party product that the team had previously adopted.

Phased implementation involved exposing application programming interfaces (APIs) to the custom application layer the team had built on top of Databricks, re-architecting how data files were being stored in the Azure Data Lake Storage (ADLS) data lake, implementing data catalogs using Apache Hive alongside Databricks SQL warehouses to query this data while stored in this data lake (rather than first needing to load another database product in order to do so), and most recently, laying the groundwork to adopt Databricks Unity Catalog integrated with Microsoft Purview.

I've been an open source champion for the bulk of my career, as I came to value the advantages that open source can provide, with the given that appropriate due diligence is performed as part of the adoption process.

Databricks is a good example of commercialized open source. In other words, open source (e.g. Apache Spark) that is bundled alongside commercially supported proprietary software, often offered by a vendor to provide additional features not otherwise available, such as infrastructure management that simplifies or alleviates the need for customers to do this work themselves.

As I mentioned previously, AI (which includes all subsets, including ML) can already be executed on Databricks, alongside what has been traditionally called "big data" processing. But while Databricks is available on all three major public clouds (AWS, Azure, and GCP), only Azure provides Databricks as a native service. In other words, Databricks is only available on AWS and GCP via third-party marketplaces, unlike on Azure. Microsoft and Databricks worked together to offer Databricks as a first-party Azure service, and Microsoft continues to stand behind this offering.

Now, make no mistake, Microsoft already offers Azure services (e.g. Azure Synapse, and on its future roadmap, Microsoft Fabric) which overlap to some extent with Azure Databricks. While there are many reasons why Microsoft likely chooses to do so, two key reasons come to mind. First, Databricks was a first mover on AWS, bringing competitive advantage that Microsoft sought to replicate, albeit with a native service. Second, Microsoft tends to cater to enterprise technology shops that are arguably often not as technically savvy as smaller firms or tech firms, and many engineers have become accustomed to the Microsoft ecosystem over the years. For example, Synapse provides C# compatibility for Apache Spark alongside Python, Scala, and R, which Microsoft thinks will lower the bar for C# engineers who prefer not learning another programming language. Of course, this means that Microsoft will always be behind on Apache Spark releases due to their needing to first convert Spark code (written in Scala) to C#, and be satisfied with their test results before the corresponding Synapse release.

I'm not surprised that Microsoft is seemingly following a somewhat similar strategy for Databricks AI (if this is to be the name, as both Microsoft and Databricks have tended to change the names of their offerings over time). OpenAI was originally open source before Microsoft got involved, and the timing of this involvement coincided with its becoming a household name seemingly overnight, with hordes of people having a generally pleasant experience after experimenting with it. As a result, Azure OpenAI was born.

But OpenAI is a black box, not trained with customer data. At least, not trained with customer data unless a given customer opts in to do so, and opting in means sharing data, something which many firms such as Deloitte seek to prevent in many use cases, such as those associated with heavily regulated domains. Of course, models trained with customer data are likely going to be the most accurate, with the given that the training performed is on par with the training that a proprietary model such as OpenAI can provide. But there are always tradeoffs to be made.

Databricks continues to follow the commercialized open source route, offering open source models that can be trained with customer data, all while keeping the data plane separate from the control plane, essentially meaning that data stays in its current location. In reference to the Databricks implementation on which my teams and I have been working at Deloitte, for example, this means that the data in the data lake doesn't move anywhere. Yes, training costs will be incurred, but again this is one of the tradeoffs that every firm looking to implement AI needs to consider. In many cases, this tradeoff may be worth the competitive advantage that is gained as a result. And based on offerings that Databricks continues to churn out for Azure customers, the additional AI tooling that Databricks will be rolling out are expected to provide additional ease of use for their target markets.

---

See all of my responses to media queries here.

Media Query Source: Part 44 InformationWeek (US digital magazine)Best way to weed out bad data"Bad data": low quality or inappropriate dataAcquire, cleanse, standardize & model data

---

My responses ended up being included in an article at InformationWeek (June 26, 2023). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The responses I provided to a media outlet on May 25, 2023:

Media: Tips for weeding out bad data.

Media: 1. What's the best way to weed out bad data? 2. What makes this technique so effective? 3. How frequently should bad data be weeded out? 4. What can happen if bad data isn't weeded out? 5. Is there anything else you would like to add?

Media: What's the best way to weed out bad data? 

Gfesser: It's important to first define what we mean by "bad" data. And perhaps just as important, we need to come to terms with what "weeding out" means. 

Bad data often means low quality data, in which case "quality" would need to be defined. But bad data can also mean inappropriate data, in which case "appropriate" would need to be defined. 

If it sounds like I'm being a stickler for definitions, it's because I am. If data is to provide any value for downstream usage, we need to help ensure that this data is reliable with respect to expected use cases.

To put this differently, not all data needs to adhere to the same strict standards because not all use cases are as demanding. And as such, judgement often needs to be used to determine what is appropriate.

 

To simplify, I consider several factors when acquiring data. First of all, from where am I sourcing the data? Does the source represent a viewpoint that I want the data to reflect when it comes time to consume it downstream? 

 

If so, what data do I actually need to acquire to meet my needs? While we can refer to selectively choosing data as "weeding out", data professionals typically refer to this step from the opposite point of view: what filtered subset of data is needed?

 

A common example of filtering often involves choosing the subset of data involving a specific date range, but any field or set of fields can be chosen. More general use cases typically involve filtering less, and more specific use cases typically involve filtering more.

 

After "landing" this data from a given source, it often needs to be cleaned to make sure it is readable by all downstream tooling, by resolving any data issues such as the presence of special characters that might get in the way.

 

After this cleansing, some data might need to be standardized so that field values are the same when representing the same thing. For example, address standardization is common.

 

In situations not involving one-offs, a common data model is typically designed to store such data so that it can be used for a variety of use cases, and as such, rules need to be applied to the data in order to ensure conformance to such a model.

 

Conversely, the design of a given model may need to be revisited if it is discovered that sourced data would be more accurately represented with alternative data structures.

 

In the world of a modern data lakehouse, all of these steps typically execute via data pipelines which acquire, clean, standardize, and structure it.

 

Media: Is there anything else you would like to add?

 

Gfesser: As data traverses a given data pipeline and is modified or restructured to meet the needs of each step or stage, data that is output from each is typically stored for reference for the purposes of visibility or traceability, otherwise known as data "lineage".

 

Data lineage is a mechanism that keeps track of data fields that have been modified along the way, as well as the code that has been used to modify it, to enable the revisiting of upstream values when understanding is needed during consumption.

 

Note that modern data lakehouses also provide the ability to store multiple stages of data in one location, via the versioning of records. As such, cleansed values of fields can be referenced, as can the "raw" values from the originally acquired data.

 

Additionally, multiple patterns of data pipelines can be constructed. For example, intermediary stages can be deleted and re-created later if the raw data and the original rules used to clean and restructure it are kept.

 

Similarly, intermediary stages don't even need to be created in some cases, for example in the case of streaming data in which steps might best be executed in memory and not persisted along the way, due to this data providing no downstream value beyond immediate consumption.

---

See all of my responses to media queries here.

Community Comment: Part 26 Discussion with purported training firm CEONormalized versus denormalized dataPerformance is important considerationMore factors involved than just record count

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6988859629183123456?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6988859629183123456%2C6989175385397608448%29

Founder & CEO of "Big Data" Training Firm:

Normalization vs Denormalization

Normalization is a process of dividing the data into multiple smaller tables with an intent to reduce data redundancy & inconsistency.

However, Denormalization is totally opposite of above idea. 
Denormalization is the technique of combining the data into a big single table. This definitely leads to redundancy in the data.

Note: Redundancy causes inconsistency -
Consider that same data is repeated 2 times and when updating you update at one place and forget to do at second place. This leads to inconsistent state.

When retrieving data from Normalized tables we need to read many tables and perform join which is a costly operation.
However, when reading the data from denormalized tables it's quite fast as no joins are required.

When to use Normalized tables vs Denormalized ones?
when we talk about OLTP systems (Online Transaction Processing) where we deal with lot of insert, delete and updates then you should go for Normalized tables.

However, when you talk about OLAP (Online Analytical Processing) systems where you need to analyse historical data then Denormalized tables are best fit. Since you wont be doing updates on data here, even though after having redundancy we wont end up in inconsistent state.

Let's take a simple Example - when you make purchase on amazon then it requires a OLTP system (a rdbms kind of database). Here the tables should be normalized.

when amazon is doing data analysis of historical data, they will create denormalized tables just to make sure analysis is faster and costly joins can be avoided.

I hope you liked the explanation, feel free to add more in comments!

What more topics you want me to cover?

PS ~ I teach big data and my students are leading big data teams in top companies. DM to know more!

#dataanalysis #dataengineering #dbms #normalization

What does this say to you?

Gfesser: One overarching theme is missing here: performance. And performance for analytical operations typically needs to be handled differently than performance for transactional operations. Why is this? Because when analytics is performed, scanning many records is often needed to calculate aggregates etc, unlike targeted queries which are looking for specific records. From my experience, it's still relatively common to hear development teams in the workplace talk about the number of records involved when it comes to performance. But there are many other additional factors. How many fields does each record contain? What queries will be run against these records? What are the data types of the field values being queried? What are the cardinalities of the field values being queried? and What database product type is being used to store this data, or conversely, What query engine / compute layer is being used, if storage is kept separate in cases of data lakes etc? All of these are important considerations.

Media Query Source: Part 43 CNBC (US business news television channel)Benefits of using self-service analytics toolsTimely, custom exploratory analyses enabledAppropriate data governance needed

The responses I provided to a media outlet on March 3, 2023:

Media: I'm working on an article about the benefits of using self-service analytics tools, and looking to include comments from an IT or data analytics executive. Only looking for executives whose organizations are using these tools.

Gfesser: When used and governed by capable hands, self-service analytics tools can provide speed of delivery for custom analyses that otherwise wouldn't be available to users. Such custom analyses are typically exploratory in nature, because the very nature of self-service implies human users. Exploring data essentially means getting familiar with data values and data relationships, and such exploration can eventually lead to discoveries of ongoing value to an organization, at which time the one-off nature of self-service analytics can be productized via automation. 

Self-service analytics tools can provide many benefits, as long as these are used and governed by capable hands. In other words, users of these tools need to understand how to use these tools as well as understand the data processed by these tools, and the data made available through these tools needs to be reliable. If any of these aspects are missing, the benefits of readily available data can actually turn into liabilities because the convenience provided to users can lead to faulty output. Think of it this way: what good does it do an organization if data is readily available to users, but either the data or the analyses performed on this data is problematic?

---

See all of my responses to media queries here.

Community Comment: Part 25 Software engineering methodologiesWaterfall versus various agile claimsPoor methodologies assume product lifespanLifespan may be too short to see success

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:7001947850091675648?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7001947850091675648%2C7001953035828563968%29

Owner at Software Delivery Firm:

What does this say to you?

#agile #waterfall #softwaredevelopment #iterativedevelopment #incrementaldevelopment

Gfesser: The first two scenarios are optimistic: these assume (1) the product actually gets past the crap stage, and (2) the lifespan of the product ends up being long enough to see this happen (in contrast with starting from scratch again).

Media Query Source: Part 42 Barron's (US financial news magazine)AI primer for financial advisorsMachine learning & deep learning: AI subsetsModel value dependent on data quality

The responses I provided to a media outlet on December 5, 2022:

Media: AI primer for financial advisors.

Media: I'm writing a post about financial advisors and AI. What is this technology? How does AI relate to areas like machine learning and deep learning? What is the importance of data? What are some things financial advisors need to know about this technology?

Gfesser: While incorrect, the terms "AI", "machine learning" and "deep learning" are often used interchangeably. AI ("artificial intelligence") is essentially about computing systems that can perform tasks typically associated with human intelligence. ML ("machine learning") is a subfield of AI which is all about task execution that relies on patterns rather than explicit instructions, and DL ("deep learning") is a type of ML based on artificial neural networks, the name of which hearkens to how the human brain filters information.

I personally don't care for the term "AI", and never use it in conversation, because it has been increasingly summoned by marketers who exaggeratingly pitch their products to what are often unsuspecting audiences, although exceptions do exist. For example, AWS ("Amazon Web Services") has acknowledged that the "AI services layer" it provides as part of its cloud offerings is really just a higher-level abstraction built on top of its ML services layer.

But data plays a crucial role regardless, because the value that AI, ML and DL provide is only as good as the data processed by it. Why is this the case? Because like any math equation, the output of a computing system's algorithms is based on the input. And if the input does not reflect "real" data, the output will likewise be unreliable. Perhaps you're familiar with the concept of GIGO ("garbage in, garbage out"). An ML algorithm or "model", for example, bases its predictions on input patterns, so if the data is problematic so will its predictions.

Financial advisors, as with other professional service providers, are expected to increasingly make use of AI over time to supplement (rather than replace) the services they provide their customers. While an advisor can generate portfolio strategies for customers without the use of AI, for example, the fact is that properly utilized AI can generate these relatively quickly with the appropriate input, additionally discovering patterns in the process that would likely otherwise go unnoticed.

---

See all of my responses to media queries here.

Product Reviews: Part 11

Two weeks ago, I reached a milestone of 850 reviews at Amazon, and I noticed for the first time that reviewer rankings have disappeared.

About 6 months ago, I had commented that my ranking had been hovering around #3000 for quite some time, and that my ranking had continued to be relatively stable since that time period, so I'm not sure what to make of this development. Some light research in the community seems to indicate that rankings have disappeared in the past, albeit seemingly for specific individuals, but the Amazon ranking web page is no longer available, and I came across a Reddit post which seems to confirm that Amazon did away with their ranking system about 5 or 6 weeks ago.

Since the milestone when my reviews reached 3500 helpful votes about 6 months ago, Amazon merchants have provided a variety of free products that I've requested from them, but apart from an elliptical machine which I ordered and then canceled (due to target family member loss of interest) and an incredible 55-inch 4k smart TV from Samsung, most items I've ordered have been extremely practical consumables: as I mentioned in my last post, this largely consists of varied nootropics and whole food vitamins.

Last year was the first in which I permitted myself to surpass the $600 threshold of free product which requires income tax reporting, keeping in mind that the vast majority of consumables don't even contribute to total product amount received. Last year was an experiment to determine the impact of surpassing the $600 threshold, and because it was negligible I decided to permit myself to do so again this year, but only for products which are either practical or borderline practical. For example, the TV replaced a 32-inch TV that I purchased 11 years ago, but I mainly ordered it for family members since I don't really watch TV, and the old TV is good enough for my movie watching while running on my treadmill.

My top review, which resulted in my shooting up to a ranking of #1967 back in May 2012, was for Madeleine Albright's new memoir at the time entitled "Prague Winter: A Personal Story of Remembrance and War, 1937-1948". At the time of that review, Amazon was still permitting downvoting in addition to upvoting, as well as commenting on reviews.

Community Post: Part 2 "Colleague", "co-worker" & "teammate"Surprising word origin of "colleague"Root word means "to arrive together"Without "conflict", there's no "colleague"

Gfesser via DALL-E: "Hieronymus Bosch painting of male and female business colleagues respectfully disagreeing with one another while running together along the Chicago lakefront."

---

A well-received community post I recently made:
https://www.linkedin.com/posts/erikgfesser_career-wordorigin-respect-activity-6998107676408770560-dHt_

Early in my #career, I called individuals with whom I worked "co-workers", and later, "teammmates", but got into the habit of calling them "colleagues" quite some time ago. I've been reading a great book called "The Ideal Executive: Why You Cannot Be One and What To Do About It", by Ichak K. Adizes, which in chapter 7 discusses the (surprising) #wordorigin of "colleague".

"What I'm going to say now may sound very simple and obvious, but it took twenty years and a lot of personal pain to discover it. I wish it *were* obvious. A colleague is not someone who agrees with you. A colleague is someone who *disagrees* with you but for whom you have #respect. Why? Because you don't learn from those who agree with you. You learn from those who *disagree*, in the course of the debate that evolves out of the conflict between you. Learning from differences is painful, but we also enrich ourselves through being different. The root of the word 'colleague', in fact, comes from the Latin word 'Collegum', which means 'to arrive together'. In other words, we started with different points of view, but through interacting we have arrived at the same point. And in Hebrew, the words Amit (colleague) and Imut (conflict) derive from the same root. Words that share the same root are interrelated; in this case, the connection is that there are no collegial relationships without conflict. And the reverse can happen: Good conflict can make people to become colleagues, if they learn from each other because and not in spite of the #conflict. Used appropriately, it can help legitimize and unite our differences. Like a spouse, who is ezer keneged (helpful against), colleagues help each other cross-fertilize ideas and expand horizons - through conflict. Without conflict, then, you don't have a #colleague. For good decision-making, team members who both respect and disagree with one another - colleagues - are essential."

After deciding to post this, I thought this might be a good occasion to experiment with the DALL-E #deeplearning image synthesis model for the first time. After generating dozens of images, I ended up choosing the attached image, which was generated by the following description I wrote: "Hieronymus Bosch painting of male and female business colleagues respectfully disagreeing with one another while running together along the Chicago lakefront." More on this in the comments below. :-)

---

Gfesser: So about the description I input into DALL-E: I ended up needing to play with the wording, and choose a variation that met my needs. My original wording simply stated "business colleagues", but all the images kept generating seemingly male images, so I explicitly added "male and female". Another interesting aspect of the generated images was that almost every single one depicted people running into each other as though they were going to collide, rather than running alongside each other, and the few which depicted people running in the same direction also had them facing forward rather than toward each other. All of this said, I'm liking DALL-E and will continue to experiment with it in the future.

Associate Professor of Computer Science and Engineering: Erik, as we know now so well, many words assume meaning with usage over time or as a matter of current convention/jargon. Most org cultures might only grudgingly accept frequent disagreements, but frequent conflicts would seem dysfunction to them. Be as that may, I think, parties need to be sure that they have facts right and POVs well-grounded. They place it respectfully as their individual submissions and do not push it beyond their powers (pay grade etc.). Some of this subject would have to do with personalities and people unwilling to take feedback/advice; [Psychologist] is a great expert here.

Gfesser: True, terms can assume different meanings over time, and as we know from the technology space, organizations can't even agree on how to define commonly used terms in the present. Adizes discusses colleagues from the perspective of the same team and same level, and argues that while leadership requires a variety of strengths, a given individual by nature cannot typically exhibit all of these, so executives should specifically look to hire those who fill these gaps: hiring individuals who repeatedly bring the same strengths to the table leads to redundancy. Per my explanation of the reason why I chose the image I posted, most of the generated images demonstrated unhealthy conflict by showing people about to collide with one another, possibly due to analysis paralysis (for lack of a better term) or different goals.

Associate Professor of Computer Science and Engineering: Erik Honestly, I've kinda firmed up away from a strengths-based approach to one that strives to match techno-project requirements of a job/assignment with my preparedness (e.g. skills, knowledge, domain etc), self-awareness & that X-factor, which definitely has to be courage, but could be grit/risk-taking/persistence as well. For example, there might not be much scientific evidence for the strengths-based approach.

Regarding the peripheral topic of etymology, off-hand I can say, the Hebrew root (Afro-asiatic family) "mit" or "mut"--which seems to mean view or thinking--have the same form "mat" & almost the same meaning in Indo-European family members--Punjabi & Hindi. So, for example, in both Hindi & Punjabi, ek-mat means unanimity, sam-mat (seh-mat in Urdu) means consensus, lok-mat means people's say, etc. Off-hand, I can't say, whether the root is Proto-Indo-European (PIE) or came across via Persian/Arabic from the Afro-asiatic family. Or the other way around--went from these Indo-European members via Persian/Arabic to Hebrew. Of interest here is that another Indo-European root sounds similar if not the same--English mate, mit (Punjabi), meet (Hindi or Urdu), mitr (Hindi/Sanskrit) all mean the same.

Gfesser: Tracing natural language lineage is always harder than doing so for computer languages, isn't it. As far as the workplace is concerned, I would agree with respect to "hard" skills, and this topic can be an especially opinionated subject due to the variety of the human experience: one-size-fits-all solutions are rare.

Associate Professor of Computer Science and Engineering: Erik True. Genomic studies seem to settle some issues there. For example, they seem to be lend credence the old steppes hypothesis that the origin of PIE is the steppes north of Black Sea & Caspian Sea :)

Personal Reading List: 4Q2022

My personal reading list for October, November, and December 2022.

Color Key: Special Notes, Completed Reading. Updated throughout the Quarter.

---

At Bat (Schnell)

---

AWS Certified Database - Specialty (DBS-C01) Certification, by Kate Gawron, Packt Publishing, 2022. Copy provided by Packt publicist Nivedita Singh.

The Ideal Executive: Why You Cannot Be One and What to Do About It, by Ichak Kalderon, The Adizes Institute, 2004.

---

 At Bat (Langsam)

---

Machine Learning Engineering in Action, by Ben Wilson, Manning, 2022. Copy provided by author Ben Wilson.

The Model Thinker: What You Need to Know to Make Data Work for You, by Scott E. Page, Basic Books, 2018.

Practical Statistics for Data Scientists: 50 Essential Concepts, by Peter Bruce and Andrew Bruce, O'Reilly, 2017. Copy provided by MarkLogic.

---

 On Deck

---

Fluent Python: Clear, Concise, and Effective Programming, by Luciano Ramalho, O'Reilly, 2015.

The Manager's Path: A Guide for Tech Leaders Navigating Growth and Change, by Camille Fournier, O'Reilly, 2017.

The Staff Engineer's Path: A Guide for Individual Contributors Navigating Growth and Change, by Tanya Reilly, O'Reilly, 2022.

---

In the Hole

---

97 Things Every Data Engineer Should Know, edited by Tobias Macey, O'Reilly, 2021. Copy provided by Promethium.

Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (Second Edition), by Christoph Molnar, Independently Published, 2022.

New Book Review: "Running with Purpose"

New book review for Running with Purpose: How Brooks Outpaced Goliath Competitors to Lead the Pack, by Jim Weber, HarperCollins Leadership, 2022:

Copy provided by Amazon.

One of the unique aspects of this book is the author's explanation of the history and purpose behind writing it in the first place. The push he needed to "commit, sit down, and write a book", in his words, came from Warren Buffett. The reason to write this book, was to tell the world about the Brooks story: "through focus, creativity, and hard work, the Brooks team has created an exceptional company, and I want more people - especially runners - to know about the brand." And from his perspective, the following are the most interesting angles about the Brooks story: (1) it's a great turnaround story, (2) it's a great David-and-Goliath, competitive strategy story, (3) it's a "purpose-driven" brand building story obsessed with runners, (4) it's a story about running with respect to the Brooks market and the author's "forty-year addiction to it", (5) it's a story about the Brooks dedication and commitment to helping managers develop into leaders, and (6) it's a story about the author's own personal journey, in which the reader might find useful lessons.

This book is broken down into 11 chapters to which Weber has interestingly associated a soundtrack, hearkening back to the music which filled his childhood home, as well as the playlists he has used during his many long runs over the years: (1) "Stumbling Out of the Blocks" - Bob Dylan's "Tangled Up in Blue" and Led Zeppelin's "Kashmir", (2) "Picking Up the Pace" - Led Zeppelin's "Ramble On", (3) "Pivot #1: Running Only" - Tom Petty and the Heartbreakers' "Running Down a Dream", (4) "Becoming an Authentic Leader" - Bob Dylan's "My Back Pages", (5) "Defining Moments: The Great Recession and an Earthquake in Running" - R.E.M.'s "It's the End of the World as We Know It", (6) "Meeting the Oracle of Omaha" - Bob Dylan's "It's Alright, Ma (I'm Only Bleeding)" and Pink Floyd's "Money", (7) "Pivot #2: Performance is Timeless (We Zig When They Zag)" - R.E.M.'s "Stand", (8) "Finding Another Gear While Navigating Global Disruptions" - Talking Heads' "Once in a Lifetime", (9) "Hitting a Wall: Fighting Cancer" - John Mayer's "Say" and Gregory Alan Isakov's "Second Chances", (10) "Filling the 'White Space' with Trust" - Joni Mitchell's "Big Yellow Taxi" and Bob Dylan's "The Times They Are a-Changin'" and "Chimes of Freedom", and (11) "On Your Left! Running Fast in a New Decade" - Macklemore and Ryan Lewis' "Can't Hold Us" and Jakob Dylan's "Something Good This Way Comes."

While as a lifelong runner I enjoyed reading the entire book, I especially liked the first five chapters comprising part 1, as well as chapter 9 and the 2-page conclusion ("Life's Short, Run Long"). After being a loyal ASICS customer for many years following a recommendation by my high school running coach, during which time I gravitated at an early point to the GEL-Nimbus shoe model, I became disappointed one year when ASICS decided to massively increase the toe box size of this model, resulting in my experimenting with a handful of brands in an attempt to find a substitute. Brooks was one of the brands I tried, and I tried several shoe models. However, after discovering Altra, a newcomer on the scene, I have been a loyal Altra customer ever since, due to its "footshape" toe box, zero drop from heel to forefoot, incredible fit, and superior traction that has met my needs as a trail runner. While I didn't particularly like any of the Brooks models I tried, one thing that has stuck with me is its "Run Happy" ethos: "when you finish your run in a happier place than when you started." I run every single morning for a reason, and this ethos sums it up quite nicely! This book caught and held my attention because it revolves around my favorite daily activity, sharing the story of one individual's journey, both personally and professionally.

Chapter 5 is an especially good read, due to the following series of sequential sections concluding the chapter: "Clinical Insight: Each Runner Has an Individual Run Signature", "Brooks's Battle Cry: Run Happy", "Bringing the Run Happy Ethos to Runners", "Performance Product + Run Happy = $1 Billion Idea", and "Running Is Too Big to Be Owned by Just One Brand." I personally wasn't aware of the paper that Brooks published on its "Run Signature" philosophy, based on years of scientific research. A biochemist studied the footfall of each of the 24 runners competing in the 10k finals of the 2012 Olympic Team Trials held at the University of Oregon. He made use of a high-speed camera on the side of the track to capture those footfalls, subsequently analyzing them side by side. What he showed is that each runner strikes the ground in a unique way, and each runner has their own personal stride, like a fingerprint. These findings informed Brooks's "Run Signature" philosophy that the only "right" way to run is the way one's body naturally wants to move. 

The two-page sidebar on what "Run Happy" is all about is in my opinion the climax of this book if a nonfiction book could have such a point. Here are a few snippets from the sidebar (numbers in this list are mine). Run Happy is (1) "a feeling, an attitude, a spirit that lives in the hearts and minds of all runners - young or old, fast or slow, male or female, master or newbie," (2) "feeling powerful, connected, passionate, energized, fast, peaceful, ecstatic, committed, free, and unstoppable," (3) passing the runner up ahead of you, in touch with something deep within, a high five with a complete stranger", "morning's first light on your favorite trail", and (4) "the best part of the runner's day, the part that understands that the worst day spent running is better than the best day at the office." With respect to this last snippet, I can personally attest that a good early morning trail run sets me up for my entire day, and enables a good night's sleep. My morning productivity getting work done at the office (whether at home or elsewhere) is extremely high for several hours following a run, as long as no unnecessary meetings get in the way.

In discussing his bout with cancer, the author surprised me by mentioning Viktor Frankl's "Man's Search for Meaning", mainly because I've just recently read two other phenomenal books related to the topics addressed by this other author: Donald Miller's "Hero on a Mission: A Path to a Meaningful Life" and Joshua Coombes's "Do Something for Nothing: Seeing Beneath the Surface of Homelessness, through the Simple Act of a Haircut". Additionally, I mentioned in my review of Jack Perconte's "The Success Trail: Learn to Win with a Marathon Runner's Mindset" that several years ago, the author of "The Purpose Driven Economy" sent me a copy of the book for my review when it was first published, and in many ways, I think Perconte's book could be realistically titled "The Purpose Driven Run." Interesting suggestion for a book title, if I do so say myself, especially when considering that Weber's book is titled "Running with Purpose", and my review of Perconte's book was published in January 2022, which is 4 months before Weber's book was published. I tend to prefer my title because it follows the pattern initially established by "The Purpose Driven Life", although I also like the fact that Weber's publisher chose to use a verb rather than a noun, and as I concluded in my review of "Hero on a MIssion", the experience of meaning requires action.

Media Query Source: Part 41 CIO (US digital magazine)Today's most notable data strategy trendData lakehouse adoptionCombination of data lake & data warehouse

The responses I provided to a media outlet on October 17, 2022:

Media: What is today's most notable data strategy trend? 

Gfesser: Data lakehouse adoption.

Media: What makes this trend so important? 

Gfesser: Data lakehouses make use of open standards to combine the traditional data lake with the traditional data warehouse, which typically resides in a separate analytical database product apart from the data lake. 

 

As such, architecture is simplified, and costs are reduced. Data does not first need to be moved from the data lake to a separate database product, and the performance of a data lakehouse can often either meet the performance of a data warehouse, or be "good enough" depending on use case. Additionally, data lakehouses enable data engineers and data scientists to work in the same collaborative environment, outside of traditional silos, enabling faster execution.

Media: Is there anything else you would like to add?

Gfesser: My current team has been working to implement a data lakehouse for Deloitte this past year, by re-architecting an existing data lake and consolidating data that was once stored in separate database products.

Implementation is being executed in phases, by addressing fundamentals first: reworking data pipelines, properly zoning data and designing data table models for this data, making use of open source Delta Lake, and sunsetting redundant cloud services.

 

In conjunction with data lakehouse adoption, we are also exploring related trends which build on this foundation, such as data sharing. While data sharing is a longer-running trend than data lakehouse adoption, which is relatively newer, we plan to continue to make use of open standards as much as possible. 

---

See all of my responses to media queries here.

Community Comment: Part 24

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6980819715766026242?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6980819715766026242%2C6980825493063245824%29&replyUrn=urn%3Ali%3Acomment%3A%28activity%3A6980819715766026242%2C6981192344935034881%29

Owner & Data Architect at Data Analytics Firm:

I love power bi, BUT

I want to break it up.

I believe getting value from data involves 2 separate parts:
- getting the data and building the data model
- analysing the data

I believe the first part (data engineering part) should be done at the central level by data experts that ensure data is consistent, well documented, and overall of high quality.

The second part (data visualization and analysis) should be done by business users .
Why?
-data viz tools are getting easier by the day
-business people understand their data better
-business people should not have to wait for someone to build their reports

Power BI seamlessly integrates the 2 parts and that is the problem.

When business people build a Power BI report:

They build data pipelines by mix-and-matching different Excel, not ensuring data quality (they don’t know how), rebuilding something that already exists, etc
Overall building low-quality foundations

When technical people build the Power BI reports:

Technical people do not know what indicators are important for the business. So they pick a good looking chart for each column in the table, and call it a day (I might exaggerate but it’s often like this)

Let’s stop asking business people to build data pipelines, and stop asking technical people to deliver insights.

Owner & Data Architect at Multiple Firms:

I tend to agree with you [Owner at Data Analytics Firm] when we are talking about creating 'SYSTEMATIC' Data / Information products (=high quality, repeatable, needs to run stable over time, agreed upon etc).

AFAIK this CAN be technically done by using SSAS as a semantic layer, right? A business-focused data engineer, let's call them 'Analytics / Semantics Engineer' would the build that conformed 'semantic layer' instead of your 'business person' persona.

On the other hand, I think it should not be 'forbidden' to create OPPORTUNISTIC data products too (one-off / ad hoc insight, no requirement to run in a stable fashion over time). If we would do so we would be back in the Business Objects / Cognos era....

Working opportunistically in PowerBI works very well!
The issue is that, as a consumer, you are often not aware whether the data products is build in a SYSTEMATIC or OPPORTUNISTIC way.

Making this distinction absolutely clear could help!

Owner & Data Architect at Multiple Firms:

Interview with [Chief Data Officer at Ministry of Justice & Security (Netherlands)] on his DQM: 
https://prudenza.typepad.com/files/english---the-data-quadrant-model-interview-ronald-damhof.pdf

Owner & Data Architect at Data Analytics Firm:

Fully agree on your distinction.

Business should not have to ask data engineering help when running an ad-hoc analysis on 3 Excels

So yes, my point is for systematic analysis.

The problem is when companies build an opportunistic data product , and they think that the product is ready to use for sistematic analysis

Then they get upset when it doesn’t work, data is bad quality, etc

It’s like building a POC and then deciding to use the POC as if it was the full product

Yes it’s already possible to build a semantic layer in Power BI (using a Power BI dataset based on SSAS technology)

Because Power BI can do everything, companies (for what I have seen) get confused on what should IT do , what should citizen developers do, etc

Owner & Data Architect at Data Analytics Firm:

[Owner & Data Architect at Multiple Firms]

Yes I have never been a fan of the ‘citizen’ version of jobs

The misbelief is that code is what separates ‘experts’ from other people

So by creating a Low-code / no-code platform, then regular people can a do lot of what experts did in the past

But coding is a small part of what separates experts. The important part is how they think: creating automatic tests, refactoring code to make it readable, using different environments and not pushing directly to production, versioning their code, and very importantly the experience

So getting rid of code gives the false belief that people can build solid applications. It’s a false belief

Gfesser: Thank you for drawing a distinction between the systematic and opportunistic. The opportunistic route can definitely be a challenging one to take, and I've seen issues surface soon after data is made available this way. Are there use cases to provide self service? Of course there are, but users need to know what they're doing, and stakeholders need to be made aware of benefits and drawbacks with this approach. And with Power BI specifically, there are benefits and drawbacks that need to be kept in mind unless it is absolutely certain that reporting is limited to ad hoc or PoC purposes. The compound challenge of course being that it's often hard to prevent this type of reporting from being repurposed as production ready. I explained in an AWS data platform use case, for example, that while it was convenient to offer users access to Power BI in Azure to hit the data in the solution we built via Power BI Gateway, this has risks including lack of visibility for a broader audience, inability to perform ad hoc querying of models from outside Power BI, and inconsistency across locally implemented data models for the same tables.
Building a Data Platform on AWS from Scratch: Part 2
https://www.linkedin.com/pulse/building-data-platform-aws-from-scratch-part-2-erik-gfesser

Owner & Data Architect at Data Analytics Firm:

Erik Gfesser

I believe you identified the real issue that is "The compound challenge of course being that it's often hard to prevent this type of reporting from being repurposed as production ready.".

Building 'opportunistic' report is not a problem. It's one of the main innovations brought by modern BI tools.
Instead of waiting for IT to build views or tables on SQL, now anyone can analyze their data mostly using drag-and-drop UI interface.

The problem is when opportunistic reports are moved to production.
Then 'exceptions' that were not considered happen in production (a join creates duplicated rows, data is missing, some data is not refreshed, etc). That is when management starts no longer trusting the reports

---

New post in reaction to comments of above post:
https://www.linkedin.com/posts/lucazanna_linkedin-ideas-activity-6981350110693572608-covS

Owner & Data Architect at Data Analytics Firm:

Best part of LinkedIn :

Sharing an idea and having people complete it / improve it / clarify it based on their experience

Thank you for the discussion [Owner & Data Architect at Multiple Firms], Erik Gfesser, [Cloud Solution Architect Data & AI at Microsoft].

Community Comment: Part 23 Architecture diagrams can often misleadDatabricks isn't a storage solutionDatabricks is a Hadoop successorBoth ETL and ELT involve input & ouput

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6968623122342195200?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6968623122342195200%2C6968866130778107904%29

Product Manager at Data Modeling Product Firm:

It's interesting how data storage technologies drive the whole data industry. Basically, companies like Databricks or Snowflake, which provide data storage solutions, create the world of data, and all other companies have to adapt their product to remain in the stream.

Gfesser: Databricks isn't a "data storage solution", and arguably, the "source" and "storage" areas of the diagram in this post are misleading. For example, why is Hadoop listed only as a potential source? Usage of Hadoop has certainly decreased significantly over time, but it's not really a source: it's technically an alternative to Databricks (after all, Hadoop is a predecessor). Perhaps most confusing are the arrows between source and storage for "ELT" and "reverse ETL". Part of me chuckles when hearing all the recent vendor talk about reverse ETL, wondering when "reverse ELT" will be introduced. Seriously, though, the source and storage areas of this diagram really need to be broken down further to help prevent misinterpretation, and a box needs to be drawn around everything to the right of the source area to delineate what is actually part of the platform. Oh, wait, perhaps I'm oversimplifying: doesn't reverse ETL involve output? :-)

Community Comment: Part 22 The phrase "it depends" still relevantDecisional factors need to be accountedNever bow to unjustified pressureYour clients & customers will thank you

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6966251443859132416?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6966251443859132416%2C6966340024552353792%29

CEO at Consulting Firm:

Software meetings best practices:

❌ Never say: Never.
✅ Always say: It depends.

Gfesser: As a consultant, I used the phrase "it depends" many times with clients. One such occasion I fondly recall involved an initial meeting with a potential client in their expansive corner office overlooking the Chicago River. In the middle of the meeting, I was asked point blank, given a choice of tech stack, what I would recommend using if building out their product. I said, "well, it depends". Their response can be summarized as follows: "Consultants still use that line? Why can't you just answer the question?" I proceeded by explaining some factors that need to be taken into account in order to make that type of decision, factors he wasn't able to address during that meeting. The result? We won the job, and I led architecture and development, writing all of the initial code. The client made their first sales to customers within a year. Never bow to unjustified pressure: your clients and customers will eventually thank you.

Community Comment: Part 21 Premature optimization is root of all evilThis maxim often taken out of contextFunctional correctness is first priorityExpected workloads impact architecture

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6952267780301754368?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6952267780301754368%2C6954395623710228480%29

CEO at ELT Platform Firm:

NEVER focus on load times when you first stand up a dashboard.

Load times DO NOT MATTER!

There are two paths you can take around load times:

1) Create a data stack that offers lightning-fast load times for insights that aren't useful.

If you take this path, you've generated no value for the business, and you've over-architected a tech stack and probably paid top dollar to do so. It's like setting money on fire in your backyard.

2) Find the metrics that really matter to your business and let them load super slowly until you find the right metrics.

If a dashboard truly offers valuable insights into how to run your business, stakeholders will WAIT. They'll sit there while the queries run because the insights are actually worth it.

Lucky for you, once you find the valuable insights (even if they load slowly), there are simple technical solutions to speed up load times:
- Indexing
- Materializations
- Streaming architecture
- Etc.

But these optimizations are a waste until you've identified the insights that are actually important to the business.

Find the valuable insights -> Then think about load times.

What other techniques do you use improve load times once you've figured out what matters?

Gfesser: Many of us were exposed to the maxim "premature optimization is the root of all evil" either during our computer science coursework or early in our software development careers. This maxim is just as important (perhaps even more so) for modern data work, which has been converging with software development in recent years. Some engineers, however, take this quote out of context. Here's the full quote by Donald Knuth: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, [they] will be wise to look critically at the critical code; but only *after* that code has been identified." Functional correctness is the first priority, but any expected work loads should obviously be taken into account because these will often have architectural impact.

Personal Reading List: 3Q2022

My personal reading list for July, August, and September 2022.

Color Key: Special Notes, Completed Reading. Updated throughout the Quarter.

---

At Bat (Schnell)

---

AWS Certified Database - Specialty (DBS-C01) Certification, by Kate Gawron, Packt Publishing, 2022. Copy provided by Packt publicist Nivedita Singh.

Machine Learning Engineering in Action, by Ben Wilson, Manning, 2022. Copy provided by author Ben Wilson.

---

 At Bat (Langsam)

---

The Model Thinker: What You Need to Know to Make Data Work for You, by Scott E. Page, Basic Books, 2018.

Practical Statistics for Data Scientists: 50 Essential Concepts, by Peter Bruce and Andrew Bruce, O'Reilly, 2017. Copy provided by MarkLogic.

---

 On Deck

---

Fluent Python: Clear, Concise, and Effective Programming, by Luciano Ramalho, O'Reilly, 2015.

Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (Second Edition), by Christoph Molnar, Independently Published, 2022.

---

In the Hole

---

97 Things Every Data Engineer Should Know, edited by Tobias Macey, O'Reilly, 2021. Copy provided by Promethium.

Presto: The Definitive Guide, by Matt Fuller, Manfred Moser, and Martin Traverso, O'Reilly, 2020. Copy provided by Starburst.

Community Comment: Part 20 Debating about the "10x" engineer continuesShifting debate to 10x interactions unwiseExcessive interaction is often a symptomSay "yes" to beneficial interaction only

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6943599797895348225?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6943599797895348225%2C6943916394010869760%29

Senior Principal Engineer at Food Delivery Firm:

Too many companies delusionally looking for that 10x engineer instead of 10x-ing interactions inside the teams.

Gfesser: Be careful to handle this philosophy with care. Increased interaction within a team isn't inherently beneficial. I've been on many teams, and some have the need for excessive interaction due to poorly planned team organization that doesn't reflect how a given product has been built out. As a longtime consultant who has typically built new products for clients, I have only seen this issue on legacy teams about to be restructured or replaced by modernization or transformation efforts by my teams and I. "Yes" to beneficial interaction only.

Product Reviews: Part 10

My ranking at Amazon continues to hover around #3000 as the 733 reviews I've written reached a milestone of 3500 helpful votes yesterday, an average of 4.775 helpful votes per review. 

Since the milestone when my reviews reached 3000 helpful votes in August 2021, Amazon merchants have provided a variety of free products that I've requested from them, with recent months including a whole house dehumidifier, large room air purifiers, and a kitchen faucet and reverse osmosis water filtration system I recently installed in my kitchen. 

While I still have this series of posts categorized under "book review", the vast majority of my reviews are no longer books, instead largely focused on practical products. Chief among these are daily supplements, the vast majority of which I now source freely from Amazon on an ongoing basis, including a variety of nootropics and whole food vitamins.

My top review, which resulted in my shooting up to a ranking of #1967 back in May 2012, was for Madeleine Albright's new memoir at the time entitled "Prague Winter: A Personal Story of Remembrance and War, 1937-1948". At the time of that review, Amazon was still permitting downvoting in addition to upvoting, as well as commenting on reviews.

Media Query Source: Part 40 CIO (US digital magazine)10 key roles every AI team needsData engineer role is foundationalSuitable data leads to trustworthiness

---

My responses ended up being included in an article at CIO (June 7, 2022). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The query responses I provided to a media outlet on May 6, 2022:

Media: I'm contributing to an article on 10 key roles every AI team needs. What are the key titles, what does each function do? (For example, data scientists, ML engineers, product managers, data engineers.) I'd love to hear from individuals holding those jobs who can briefly talk about what their roles entail, or enterprise AI managers who manage these teams.

Gfesser: I'll speak to "data engineer", a role I've served in the past and a role for which I currently hire and manage, although I have served in other related roles as well.

While the roles of data engineer and ML engineer are closely related, I consider the data engineer foundational for both ML and non-ML initiatives.

In a nutshell, data engineers build data pipelines to collect and assemble data for downstream usage, and in a DevOps world, additionally build pipelines to implement the infrastructure on which these data pipelines run.

For example, when implementing data pipelines in one of the public clouds, a data engineer needs to first write the scripts to spin up the necessary cloud services which provide the compute necessary to process ingested data.

This ingested data is then transformed and subsequently stored in increasing states of consumption readiness, which can then be consumed for a variety of uses, including ML.

Data engineers are typically required to be skilled in languages such as Python, SQL, and Scala, libraries used for cleansing and transforming data, and when large volumes of data are involved, de facto standard Apache Spark for distributed data processing.

Data engineers are considered foundational because data needs to be collected and made suitable for consumption before anything trustworthy can be done with it.

Note that the role of data engineer has evolved over the last several years to a point that, not only is it considered critical to all serious data initiatives, it is also converging with other roles needed by ML teams.

Instead of waiting for a data engineer to build what is necessary for a data consumption use case, for example, other roles such as data scientist can instead do this work themselves, or vice versa.

That said, it is typically easier for a given skilled data engineer to pick up data science skills, due to their presumed software development background and because industry has not placed enough attention on building data engineering skills relative to data science.

---

See all of my responses to media queries here.

Community Comment: Part 19 Firms have more choices than vendors admitThe "buy versus build" choice is nuancedCommercial & open source fall on spectrumBlanket $ statements cannot be made

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6926625786309316608?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6926625786309316608%2C6926689203154018305%29

CEO at ELT platform firm:

Do you think your data team is special?
It’s not.
You take one of two approaches.

Strategy 1. Invest in people in service of open source software.
Strategy 2. Invest in SaaS software in service of people.
Your approach directly determines who you hire, how you spend money, and which technologies you use.

Let me outline the two approaches in more detail.

Approach 1.
Teams that invest in people in service of open source software.
This is common in a handful of situations:
- When data teams are born out of engineering teams.
- Data teams within particular industry verticals - like healthcare and financial services.
- Very large enterprises (i.e. the Fortune 500).
- Non-US companies (where SaaS software for data management is less accessible).
These teams invest heavily in headcount.
The biggest line item for the data team will be engineering talent on staff.

Approach 2.
Teams that invest in SaaS software in service of people.
This is common in a handful of scenarios:
- When data teams are built out under a business function.
- Data teams within high-growth industries, with lighter weight procurement processes (eCommerce, marketing, etc.).
- Mid-market companies with lean teams.
- US-based companies.
These teams invest heavily in SaaS software.
As a result, budgets are more evenly split between headcount (typically very lean teams) and SaaS software.

How can you tell which type of team you’re building?
There are two easy tests:
1. When you see a problem and need a solution, do you first try to build in-house? Or do you first try to find a SaaS solution?
2. Is your team primarily budgeting for headcount or software?

I only came to this realization after I started selling to data teams.
Why?
As a vendor in the data world, you start by thinking all data teams are the same.
At [ELT platform firm], we’re a SaaS solution.
We’ve realized we’re not a good fit for teams that invest in people in service of open source software.
The idea of leveraging a SaaS solution goes against the culture, the procurement methodology, and the architecture of those companies.
Have other vendors arrived at the same realization?

For those people in analytics.
Which type of data team are you building?

Gfesser: Interesting thoughts. In the context of your argument, I would typically call strategy #1 "invest in people in service of custom development" because the opposite of buying software is building software, and making use of open source tech is one of the options for construction. Tech savvy firms will obviously make heavy use of open source tech. But there is something else to consider that I often see missing in discussions such as this: buying commercialized open source. In other words, commercial software which has been built on open source. AWS and GCP, for example, provide quite a few examples of such offerings. But I don't think the quantities of talent for these two strategies are at opposite ends of a spectrum.