Erik on Software

Community Comment: Part 30 Databricks Unity Catalog & Microsoft PurviewNot an either-or proposition for AzureLikely need to integrate these productsIntegration dependant on several factors

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:ugcPost:7097673607933292544?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7097673607933292544%2C7098975094026207232%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287098975094026207232%2Curn%3Ali%3AugcPost%3A7097673607933292544%29

Cloud Solutions Architect at Consulting Firm:

Read my latest post if you want to know what I think.

Best data governance tool: Databricks Unity Catalog or Microsoft Purview?
https://www.linkedin.com/pulse/best-data-governance-tool-databricks-unity-catalog-matias-samblancat/?trackingId=3yW1GpYsQby6qEc5ZzJkbQ%3D%3D

Gfesser: I'm currently working on implementing both Unity Catalog *and* Microsoft Purview for our data platform with my team. This article provides a decent summary, but as I agree that these products are moving fast, pay particular attention to the concluding "key considerations" section. Also, keep in mind that this article focuses more on differences between these two products, rather than integrating these together. The closing paragraphs mention that "we know [MS] is working on a new connector to streamline the process and hope [it] is ready to use soon", but a connector is already available, albeit still in preview state, and as my team cannot make use of non-GA products in production, I'm working with MS to get a better sense of the ETA. Note also that MS implemented an OSS accelerator to integrate these, but both Databricks *and* MS have advised my not using it due to tight coupling with specific library versions of Spark etc. Also keep in mind that (1) regarding data lineage etc, UC scope is constrained to data and operations against this data registered with UC, whereas Purview has greater purview (pun intended), and (2) you may want to consider using Purview if your architecture needs to consider the longer term Fabric roadmap.

Cloud Solutions Architect at Consulting Firm:

very good insight, thanks for sharing your experience. I agree that including Fabric into the technology stack might change things slightly, specially for companies planning to move workloads from dbr to Fabric (if any?). So far, Dabricks did not have strong competition in the data engineering space, but will Fabric change that? To be seen.. If that happens, it might make sense for those companies to avoid being Databricks centric organization but I believe it will have to be decided case by case as every context is different.

---

Cloud Solutions Architect at Toyota:

Thanks for sharing, [Cloud Solutions Architect at Consulting Firm].

I agree with the separation that Unity is more technical facing than Purview. But Apache Atlas API capability of Purview is still tough to beat.

Gfesser: Completely agree. Microsoft was wise to not only make use of OSS under the covers, but expose its APIs. 

---

IT Manager & Data Governance Product Owner at German Real Estate Firm:

Currently, there are NO data quality features for Purview which are in private preview or generally available.

Gfesser: That's correct. I see us continuing to make use of custom code and Great Expectations for the data quality components of our data platform.

Community Comment: Part 29 Normalized versus denormalized dataData storage dependent on usage patternsDenormalized data makes reads efficientNormalized data makes writes efficient

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:7102716601111560192?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7102716601111560192%2C7102948085684740096%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287102948085684740096%2Curn%3Ali%3Aactivity%3A7102716601111560192%29

CEO at Business Intelligence Product Firm:

After 100s of conversations with analytics engineers, I'm convinced the way we build data pipelines needs to change.

We need a simpler, lower-effort way to build our data models. That's why I'm suggesting NERD modeling - Narrow, Entity-focused, Relational, and DRY:

The four pillars of NERD modeling:

Narrow:
Data teams are finally realizing they have too many models.

Recent...pricing discussions have been a trigger point for people to trim down. But there's other reasons: larger data pipelines have more places to break, are less maintainable, and are harder to understand.

Keep data pipelines small by keeping them short. Minimize the number of models between the raw ingested data and the BI tool. If you have 5 steps to prepare and stage, you're probably doing it wrong.

Entity-focused:
Maxime Beauchemin recently introduced ECM (link below).

The Entity-Centric idea is brilliant. It means aggregating fact features up into the dimension tables. For eg. in your users table, you'd pre-calculate features such as 'logins_last_90_days.'

It's brilliant because:
(a) thinking about features by entity type is an intuitive way to consume data ('total five-star reviews by product'),
(b) this gives you a huge surface area to discover new insights. Every analytics project has inputs and outputs. What features drive which outcomes? Adding in entity features gives you a ton of new inputs to use.

You may have noticed this looks like Feature Engineering for machine learning. That's true. And since it works great for ML models, it's also very helpful for exploring data as a human.

But now - one area where NERD differs: it is...

Relational:
Lately everyone has been ditching facts/dims in favor of One Big Table (OBT). This makes sense if you haven't set up a semantic layer yet.

But as more people switch to semantic layers, it makes more sense to leave the final 'Consumption' tables as relational facts and dimensions. Then, your semantic layer prepares more manageable tables dynamically with the metrics you need.

There's several reasons to keep your consumption-level data in facts and dimensions. It keeps your data models small, narrow (see N), and composable. It's intuitively understandable (especially when the fact tables are all Entities - see E).

And perhaps most imporantly - relational data models are great because they're...

DRY!
Don't Repeat Yourself.

You never have just 1 One Big Table. You have dozens of them, with partially overlapping metrics. This is a recipe for chaos.

If your end users need to know churn, they don't know if they should use the users_behavior_new OBT, or the retention_calculation_v3 table. If you need to change your definition of churn, you'll do it repeatedly, in many poorly-referenced places.

By basing metrics directly off of the same fact table, you do it exactly once.

That's the NERD modeling philosophy. If people want a playbook on how to do it, lmk in comments and I'll write a blog post.

Gfesser: The manner in which data is stored should be dependent on planned usage of that data. The concept of "one big table" is an old concept: it's called "denormalized" data. And in order to understand denormalized data, it helps to understand normalized data. I'm not sure I see these concepts explained here. When data is normalized, it essentially means there are no relationships within a given table beyond the one that exists between the key for that table and the rest of the columns. This makes writes efficient. However, when data is denormalized it essentially means that data that would otherwise be normalized across multiple tables is stored in one table, meaning that, yes, there are additional relationships in that combined table. This makes reads efficient. Why? Because data is pre-joined: read queries don't need to waste time joining tables together at the time of consumption: the work to do this has already been done beforehand. And this is why usage patterns matter. The data in this denormalized, one big table typically can't serve all needs: the manner in which this data has been combined has already taken into account what queries will be needed to run against it.

CEO at Business Intelligence Product Firm:

Erik Gfesser Thanks Erik and fun fact: I’d actually
considered using Normalized for the N because I agree it’s valuable.

My philosophy on this is that strict 3NF usually isn’t necessary and you can store low-cardinality items directly in a dimensional table. But normalized is definitely my preferrence.

Yep, you’ll absolutely need to do more dynamic joins with it (and some huge tables you may need to denormalize - eg events). But most of those joins are small and it unlocks a lot more flexibility/detail in the analytics.

And normalized is DRYer!

Gfesser: One of the key takeaways for everyone reading this post: a "relational" table doesn't imply third normal form, because relational tables can exhibit any of several degrees of normalization. And yes, from another perspective one could instead choose to say several degrees of denormalization. All that "relational" means is that relationships exist: it doesn't say anything about the nature of these relationships.

Community Comment: Part 28 Discussion with purported CTOsNot all technical debt is the sameTechnical debt is subset of organizational debtBusiness + technology = "our business"

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:7105460146390089728?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7105460146390089728%2C7105524357036179456%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287105524357036179456%2Curn%3Ali%3Aactivity%3A7105460146390089728%29

Fractional CTO and "Engineering Management Expert":

"Technical debt? Is that a joke?"

CEOs and non-technical CxOs should not have to hear about technical debt — if the engineering department manages it well.¹

Technical debt is is a measure of these quantities:
- DUCT TAPE that holds your system together.
- RUST that has accumulated.

Duct tape represents improvisations resulting from:
- Quick fixes that were needed to "plug leaks".
- Pressure to produce too fast, and not doing things properly.
- Implementing experimental features with minimum effort, in order to gather data to decide whether to pursue them.

Rust: 
Systems slowly decay, as if they are rusting.
For example:
- Core assumptions are eroded as the business evolves.
- Third party APIs and libraries evolve, requiring changes.
- Performance is degraded due to significant traffic growth.
- Security best practices change and patches need to be handled.

SO WHAT?!

Advantages of incurring temporary tech debt:

😎 Faster feature implementations. This is worthwhile because 85% of features fail to deliver the promised value. Experiments that don't make it can just be removed. Successful experiments will need a second implementation phase.

😎 Faster MVP implementation. Just like for features, only this is the whole product.

Disadvantages of carrying long term tech debt:

😡 Lost productivity — the more improvisations there are, the longer it takes to deliver new features. You could call this "interest" on the debt.

😡 Lost business — due to breakdowns, missing features, bad performance, security issues.

😡 Increased employee churn — nobody wants to work on a system that is not well maintained, has frequent breakdowns and emergencies, with little hope of being able to change this.


What is your company's attitude to technical debt?

Gfesser: Hi [Fractional CTO and "Engineering Management Expert"], good post. I'll appreciate your explaining what the intersection of the y-axis and x-axis is intended to represent on your diagram. Does this mean coordinates of 0,0? If so, what does it mean when there is negative cost? Perhaps what you're looking to convey here is that there are cost gains until a hypothetical threshold is reached, at which point these gains start getting reverted. Also, what is your view with respect to whether cost, fragility, and trouble are always directly correlated with each other? Since there is no time axis on this diagram, might it be likely that cost increases in the short-term as tech debt is being paid down? If so, what have you personally found to be a good strategy communicating the benefits and drawbacks of this short-term dilemma?

Fractional CTO & "Engineering Management Expert":

Great — someone is paying attention! Yes Erik, I'm trying to indicate that a small amount of tech debt is beneficial, especially if it's because of experimentation. The most intuitive way for me to show that was to move the graph down below zero. But your question makes me understand that I need to think about this a bit more, and label the origin and y axis clearly and explicitly. The same goes for your question on why I grouped cost, fragility and trouble. I was tempted to label it just "trouble". As I said, I need to ponder on this for a while — thanks for raising these points!

---

Fractional CTO and Executive and Technology Leadership Coach:

Given the source of technical debt is usually business decisions, I’ve come to the conclusion that technical debt is a myth (or a misnomer, if you want to be precise).
https://medium.com/@nrcantor/technical-debt-is-a-myth-cb122d3c2717

Gfesser: [Fractional CTO and Executive and Technology Leadership Coach] I think you've made some good points in your article, regardless of whether I agree with everything you've stated. In particular, I like your raising the concept of an "organizational mortgage". I've consulted dozens of firms over the course of my career, and this is a very real issue that many organizations face. Put another way, many firms have organizational debt just like many of these same firms have technical debt, but I would argue that (1) the technical debt these firms possess is really a subset of their overall organizational debt, and (2) many of these same firms unfortunately choose not to look at their issues from a holistic perspective. One of the challenges in discussing this dilemma, however, is that many firms, regardless of domain, are now arguably software firms often run by individuals who don't have software development backgrounds, and as such, systemic issues surface in the software that they build. There is such an overabundance of issues in many firms that it's oftentimes not realistic to single out individual groups, so I continue to hold the view that it's healthier for a given organization to refer to "our business" rather than referring to business and technology as separate entities.

Fractional CTO and Executive and Technology Leadership Coach:

Erik Gfesser You’re absolutely right about technical debt being a subset of organisational debt. They go hand in hand, and the same thinking drives the accumulation of both. Just like organisational debt, it’s often the right decision, even though the consequences are differencent off they’re ignored.
One of the perspective I try to hold I’d the same as what you’ve described - technology and the business aren’t separate entities. They’re intimately connected, interdependent parts of a whole. If I’m completely honest, however, I sometimes struggle to keep that perspective, given how many years I was on ‘one side’ of the imaginary divide.
I think the companies that successfully adopt a holistic, system-wide view, have a massive competitive advantage, because they can see the consequences of decisions in ways that functionally divided companies can’t.

Community Comment: Part 27 Discussion with purported staff data engineerTransactional versus analytical modelingMaking broad generalizations doesn't helpQuoting $ savings in vacuum not advised

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:7116120255428493312?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7116120255428493312%2C7116135281384202240%29&replyUrn=urn%3Ali%3Acomment%3A%28activity%3A7116120255428493312%2C7116171801327517696%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287116135281384202240%2Curn%3Ali%3Aactivity%3A7116120255428493312%29&dashReplyUrn=urn%3Ali%3Afsd_comment%3A%287116171801327517696%2Curn%3Ali%3Aactivity%3A7116120255428493312%29

Owner of Data Engineering Training Firm and former Staff Data Engineer at Airbnb:

If you model your data for analytics and you're doing transactions, you're going to have a bad time.

If you model your data for transactions and you're doing analytics you're going to have a bad time.

Why is this the case?

Transactional data modeling (OLTP) is optimized for access to data of a single user or entity.

Analytical data modeling (OLAP) is optimized for accessing the entire data set.

If you have an OLTP use case and you model OLAP, you'll get excess IO and unnecessary data for your transaction.

If you have an OLAP use case and you model it OLTP, you'll get excess compute and JOINs and shuffle your data all over the place.

Most online systems are OLTP. Most data engineers' jobs are taking that OLTP model and converting it into a robust OLAP system for data scientists.

#dataengineering

Developer Advocate at Cloud Data Warehouse Software Firm:

That used to be the case, but isn't anymore.

Highly normalized structures work perfectly fine in modern data warehouse platforms. Joins are of little cost anymore.

This reduces the necessity to transform due to physical limitations and can decrease time to market for any given project by up to 40%

Owner of Data Engineering Training Firm and former Staff Data Engineer at Airbnb:

I don't believe this is true and sounds more like marketing than reality.

For example, although you'll call this an edge case probably, it took over 20 database tables joined together to determine the price and availability of a given listing on a given night at Airbnb.

You're saying there's no value in making an analytical table that people can query since they can just do the joins themselves right?

Those joins took 15 minutes to run.

The materialized tables I gave them gave an answer in <2 seconds.

Gfesser: Both of you are making very broad generalizations without mentioning all the factors that come into play. And while both of you are correct for some use cases, quoting time-to-market savings is always ill advised in a vacuum: this should never be done until after an assessment has been performed.

New Book Review: "Running Toward Life"

New book review for Running Toward Life: Finding Community and Wisdom in the Distances We Run, by John Trent, Broad Book Press, 2023:

Copy provided by Amazon.

Trent mentions in his introduction that his hope in writing this book is to speak to various phases of one's running life, so that readers will walk away with a similar sense of understanding as to who they are and what is important to them. These phases, which correspond to the three parts of this book, are "Finding Your Mentors", "Choosing Your Companions", and "Overcoming Your Challenges". He goes on to explain that "although these stories are highly personal, they contain some of the larger truths that we all experience in our lives. That we are constantly learning and evolving. That as time passes and we accrue thousands of hard-earned miles on the trails, there are important lessons that are equally hard-earned and can stick with us if we allow them to live on. And that as we gain these experiences, we come to view them more like a way station and not a final destination, where we have the opportunity to continually ask ourselves important questions that help us grow and learn. No matter what distance of ultra we run, we come to understand that whenever we go the distance, it isn't just the number of miles that we have run that matter. As I found out in 1995, you can run an ultra distance that comes on the periphery of something larger and it may not ever be remembered - except by you. Running ultras is a way of running toward what matters most: understanding and finding a greater meaning in life that we live. It helps us run toward our lives."

The closing paragraph of chapter 11 ("Your Days Are Short Here"), which is the last chapter of the "Overcoming Your Challenges" part of this book, echoes the title, but it seems the last sentence quoted above interprets "life" differently. "The dreams we possess we don't ever actually own. If they are to be true dreams, dreams that matter, we must share them with others. Whenever we do, the races that we run will never seem as long. Whenever we do, we guarantee that we are always running toward something that truly matters. Whenever we run, we should always run toward life." Running toward life implies that life exists outside oneself, because running wouldn't be needed if it exists within. As such, this means something spiritual to me. In contrast, Trent's earlier discussion didn't mention running toward life, but toward one's own life. In my view, it makes more sense for a given human being to strive for something they don't already possess. For example, 1 Corinthians 9:24-27 puts it this way: "Do you not know that in a race all the runners run, but only one gets the prize? Run in such a way as to get the prize. Everyone who competes in the games goes into strict training. They do it to get a crown that will not last, but we do it to get a crown that will last forever. Therefore I do not run like someone running aimlessly; I do not fight like a boxer beating the air. No, I strike a blow to my body and make it my slave so that after I have preached to others, I myself will not be disqualified for the prize."

That said, I like that Trent discusses ultra running so eloquently as a social activity that brings people together to accomplish a goal. The goal he discusses seems to be actually summarized best in the subtitle of this book: "Finding Community and Wisdom in the Distances We Run." However, this is elaborated upon a bit better in chapter 7 ("The Truths of Wasatch"): "As runners, we start out chasing the things we wish to conquer - time and distance. We wish to run as fast as we can, for as long as we can. Even if we slow, and time becomes less of a motivator, covering the sheer ultra marathon distances remain our primary intoxicant. As I've grown older, and as my body has told me in no uncertain terms that it is acutely aware of what I am trying to do to it, the allure has been less about time and distance and more about every human's wish to remain connected with other people." Trent later recalls a book he read by Roger Angell called "This Old Man", which he considers a deeply moving look at growing old. "He wrote of our basic need as we get older to know that as we live, we always crave companionship to make it come to life: 'Getting old is the second-biggest surprise of my life, but the first, by a mile, is our unceasing need for deep attachment and intimate love.' "

 

As a lifelong runner who started as a student athlete and continues to run on a daily basis, running hasn't been much of a social activity for me because I almost always run alone. While I initially ran alongside other student athletes over a ten year time period, apart from the relay races I ran my times were always recorded for me as an individual. Ultra racing typically works the same way, but much of what Trent shares in this book is the camaraderie that forms between individuals as they run long distances, and this is largely a foreign concept for me because even when training for the marathon one year I found that I'm not well suited for lengthy conversations while running. I mentioned in my review of a book by Vanessa Zuisei Goddard called "Still Running: The Art of Meditation in Motion" that "I personally use a mantra as the point of focus while running, and when I do this I am actually doing so to zone out...In fact, I've often commented to others that time seemingly passes by much more quickly when I do this. Now, thinking back to my beginnings as a runner in grade school, I was explicitly coached to focus on my breathing. However, once I got this down I found it distracting to be consciously aware of my breathing, so over time removed this practice from my repertoire." As someone inexperienced with ultra running, talking would seem to be exceedingly more distracting than paying attention to my own breathing. I don't race as an adult because I got burned out as a student athlete, but in retrospect the contrast is clear: my races were largely selfish, and with the right intent one could arguably reword the title of this book from "Running Toward Life" to "Running Toward Selflessness".