Erik on Software

Media Query Source: Part 45

---

My responses ended up being included in an article at Reworked (September 7, 2023). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The responses I provided to a media outlet on August 29, 2023:

Media: What does Databricks do and what can it bring to Microsoft Azure?

Media: Recently, Microsoft announced that it is looking at offering Databricks AI through Azure. This would allowMicrosoft to offer Databricks' machine learning (ML) and data analytics tools to Azure customers as part of its cloud services.

Databricks provides an AI-powered data analytics platform to help organizations build their own AI models.

However, this deal contrasts with OpenAI's approach of developing proprietary AI models and licensing them to partners like Microsoft. The two companies work closely on Microsoft 365, Windows, Bing and more.

By offering Databricks' software on Azure, Microsoft aims to meet the growing enterprise demand for customized AI tools. This allows companies to leverage AI in applications tailored to their specific business needs.

Gfesser: I first made use of Databricks in 2018, since the time it was made generally available (GA) in Azure.

My first experience while with SPR Consulting involved addressing all of an insurance client's issues in their unsuccessful attempts to adopt Databricks. This work included building a slew of proofs-of-concept (PoCs) demonstrating how to do everything they were looking to do, followed by migrating their machine learning (ML) models to Databricks to run a pilot, as well as providing recommendations on how team processes were being executed and how data pipelines were designed.

While with Deloitte these past two years, I've driven several initiatives that started with the roadmap I laid out for phased implementation of a data lakehouse with Databricks. While Databricks was already being used in production, few knew this because it had been solely implemented as a back-end engine to execute the workloads sent to it by a non-Azure third party product that the team had previously adopted.

Phased implementation involved exposing application programming interfaces (APIs) to the custom application layer the team had built on top of Databricks, re-architecting how data files were being stored in the Azure Data Lake Storage (ADLS) data lake, implementing data catalogs using Apache Hive alongside Databricks SQL warehouses to query this data while stored in this data lake (rather than first needing to load another database product in order to do so), and most recently, laying the groundwork to adopt Databricks Unity Catalog integrated with Microsoft Purview.

I've been an open source champion for the bulk of my career, as I came to value the advantages that open source can provide, with the given that appropriate due diligence is performed as part of the adoption process.

Databricks is a good example of commercialized open source. In other words, open source (e.g. Apache Spark) that is bundled alongside commercially supported proprietary software, often offered by a vendor to provide additional features not otherwise available, such as infrastructure management that simplifies or alleviates the need for customers to do this work themselves.

As I mentioned previously, AI (which includes all subsets, including ML) can already be executed on Databricks, alongside what has been traditionally called "big data" processing. But while Databricks is available on all three major public clouds (AWS, Azure, and GCP), only Azure provides Databricks as a native service. In other words, Databricks is only available on AWS and GCP via third-party marketplaces, unlike on Azure. Microsoft and Databricks worked together to offer Databricks as a first-party Azure service, and Microsoft continues to stand behind this offering.

Now, make no mistake, Microsoft already offers Azure services (e.g. Azure Synapse, and on its future roadmap, Microsoft Fabric) which overlap to some extent with Azure Databricks. While there are many reasons why Microsoft likely chooses to do so, two key reasons come to mind. First, Databricks was a first mover on AWS, bringing competitive advantage that Microsoft sought to replicate, albeit with a native service. Second, Microsoft tends to cater to enterprise technology shops that are arguably often not as technically savvy as smaller firms or tech firms, and many engineers have become accustomed to the Microsoft ecosystem over the years. For example, Synapse provides C# compatibility for Apache Spark alongside Python, Scala, and R, which Microsoft thinks will lower the bar for C# engineers who prefer not learning another programming language. Of course, this means that Microsoft will always be behind on Apache Spark releases due to their needing to first convert Spark code (written in Scala) to C#, and be satisfied with their test results before the corresponding Synapse release.

I'm not surprised that Microsoft is seemingly following a somewhat similar strategy for Databricks AI (if this is to be the name, as both Microsoft and Databricks have tended to change the names of their offerings over time). OpenAI was originally open source before Microsoft got involved, and the timing of this involvement coincided with its becoming a household name seemingly overnight, with hordes of people having a generally pleasant experience after experimenting with it. As a result, Azure OpenAI was born.

But OpenAI is a black box, not trained with customer data. At least, not trained with customer data unless a given customer opts in to do so, and opting in means sharing data, something which many firms such as Deloitte seek to prevent in many use cases, such as those associated with heavily regulated domains. Of course, models trained with customer data are likely going to be the most accurate, with the given that the training performed is on par with the training that a proprietary model such as OpenAI can provide. But there are always tradeoffs to be made.

Databricks continues to follow the commercialized open source route, offering open source models that can be trained with customer data, all while keeping the data plane separate from the control plane, essentially meaning that data stays in its current location. In reference to the Databricks implementation on which my teams and I have been working at Deloitte, for example, this means that the data in the data lake doesn't move anywhere. Yes, training costs will be incurred, but again this is one of the tradeoffs that every firm looking to implement AI needs to consider. In many cases, this tradeoff may be worth the competitive advantage that is gained as a result. And based on offerings that Databricks continues to churn out for Azure customers, the additional AI tooling that Databricks will be rolling out are expected to provide additional ease of use for their target markets.

---

See all of my responses to media queries here.

Media Query Source: Part 41

The responses I provided to a media outlet on October 17, 2022:

Media: What is today's most notable data strategy trend? 

Gfesser: Data lakehouse adoption.

Media: What makes this trend so important? 

Gfesser: Data lakehouses make use of open standards to combine the traditional data lake with the traditional data warehouse, which typically resides in a separate analytical database product apart from the data lake. 

 

As such, architecture is simplified, and costs are reduced. Data does not first need to be moved from the data lake to a separate database product, and the performance of a data lakehouse can often either meet the performance of a data warehouse, or be "good enough" depending on use case. Additionally, data lakehouses enable data engineers and data scientists to work in the same collaborative environment, outside of traditional silos, enabling faster execution.

Media: Is there anything else you would like to add?

Gfesser: My current team has been working to implement a data lakehouse for Deloitte this past year, by re-architecting an existing data lake and consolidating data that was once stored in separate database products.

Implementation is being executed in phases, by addressing fundamentals first: reworking data pipelines, properly zoning data and designing data table models for this data, making use of open source Delta Lake, and sunsetting redundant cloud services.

 

In conjunction with data lakehouse adoption, we are also exploring related trends which build on this foundation, such as data sharing. While data sharing is a longer-running trend than data lakehouse adoption, which is relatively newer, we plan to continue to make use of open standards as much as possible. 

---

See all of my responses to media queries here.

Community Comment: Part 23

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6968623122342195200?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6968623122342195200%2C6968866130778107904%29

Product Manager at Data Modeling Product Firm:

It's interesting how data storage technologies drive the whole data industry. Basically, companies like Databricks or Snowflake, which provide data storage solutions, create the world of data, and all other companies have to adapt their product to remain in the stream.

Gfesser: Databricks isn't a "data storage solution", and arguably, the "source" and "storage" areas of the diagram in this post are misleading. For example, why is Hadoop listed only as a potential source? Usage of Hadoop has certainly decreased significantly over time, but it's not really a source: it's technically an alternative to Databricks (after all, Hadoop is a predecessor). Perhaps most confusing are the arrows between source and storage for "ELT" and "reverse ETL". Part of me chuckles when hearing all the recent vendor talk about reverse ETL, wondering when "reverse ELT" will be introduced. Seriously, though, the source and storage areas of this diagram really need to be broken down further to help prevent misinterpretation, and a box needs to be drawn around everything to the right of the source area to delineate what is actually part of the platform. Oh, wait, perhaps I'm oversimplifying: doesn't reverse ETL involve output? :-)

Community Comment: Part 19

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6926625786309316608?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6926625786309316608%2C6926689203154018305%29

CEO at ELT platform firm:

Do you think your data team is special?
It’s not.
You take one of two approaches.

Strategy 1. Invest in people in service of open source software.
Strategy 2. Invest in SaaS software in service of people.
Your approach directly determines who you hire, how you spend money, and which technologies you use.

Let me outline the two approaches in more detail.

Approach 1.
Teams that invest in people in service of open source software.
This is common in a handful of situations:
- When data teams are born out of engineering teams.
- Data teams within particular industry verticals - like healthcare and financial services.
- Very large enterprises (i.e. the Fortune 500).
- Non-US companies (where SaaS software for data management is less accessible).
These teams invest heavily in headcount.
The biggest line item for the data team will be engineering talent on staff.

Approach 2.
Teams that invest in SaaS software in service of people.
This is common in a handful of scenarios:
- When data teams are built out under a business function.
- Data teams within high-growth industries, with lighter weight procurement processes (eCommerce, marketing, etc.).
- Mid-market companies with lean teams.
- US-based companies.
These teams invest heavily in SaaS software.
As a result, budgets are more evenly split between headcount (typically very lean teams) and SaaS software.

How can you tell which type of team you’re building?
There are two easy tests:
1. When you see a problem and need a solution, do you first try to build in-house? Or do you first try to find a SaaS solution?
2. Is your team primarily budgeting for headcount or software?

I only came to this realization after I started selling to data teams.
Why?
As a vendor in the data world, you start by thinking all data teams are the same.
At [ELT platform firm], we’re a SaaS solution.
We’ve realized we’re not a good fit for teams that invest in people in service of open source software.
The idea of leveraging a SaaS solution goes against the culture, the procurement methodology, and the architecture of those companies.
Have other vendors arrived at the same realization?

For those people in analytics.
Which type of data team are you building?

Gfesser: Interesting thoughts. In the context of your argument, I would typically call strategy #1 "invest in people in service of custom development" because the opposite of buying software is building software, and making use of open source tech is one of the options for construction. Tech savvy firms will obviously make heavy use of open source tech. But there is something else to consider that I often see missing in discussions such as this: buying commercialized open source. In other words, commercial software which has been built on open source. AWS and GCP, for example, provide quite a few examples of such offerings. But I don't think the quantities of talent for these two strategies are at opposite ends of a spectrum.

Community Post: Part 1 (Open Letter to DB-Engines)

---

Background for a community post I wrote:
https://www.linkedin.com/posts/erikgfesser_gartner-databricks-database-activity-6925028958199570432-t5bW/

Gfesser: Success! About a week ago, I submitted the following comment on a "DBMS Market Share Ranks" post made by a VP at Gartner. After subsequently following up with db-engines, they have just informed me that, after careful consideration, my proposal to add Databricks to their monthly ranking has been approved, and it will be included starting in May 2022 (link to my original post in the comments below). In the meantime, Databricks has been added to their catalog of 391 database products (link to this catalog in the comments below).

New db-engines entry for Databricks:
https://db-engines.com/en/system/Databricks

---

My open letter to the folks at db-engines on April 20, 2022:

Gfesser:

I've been a heavy user of db-engines over the years, and continue to appreciate the information it provides.

I last contacted you in June 2017, and had some brief exchanges with Dr Paul Andlinger.

The other day, I visited db-engines for the first time in a while, and noticed that Databricks isn't yet represented in the rankings. 

I'll greatly appreciate your explaining why this is the case, as well as your considering it being added.

The other day, I had a conversation on LinkedIn about the latest Gartner ranking on database vendors, and I noticed that Snowflake is present in both the Gartner and db-engines rankings, but Databricks is only present in the Gartner ranking.

I told the community that I would follow up with you on this.

In recent months, I've introduced use of Databricks SQL to a data platform for which I serve as chief architect. Storage and compute are separate, with compute provided by Databricks (Spark) clusters and storage provided by the data lake. SQL queries can be executed against any data for which Hive tables have been created.

Based on db-engines database categorizations, it looks like Databricks would fit in the "relational DBMS" category. 

Other similar products I've used, such as Presto and Impala, are also represented in this category. You might also want to consider adding Amazon Athena to this category at the same time, which is built on Presto. 

Also, I happened to notice in the "search engine" category that you've included "Microsoft Azure Search": the name of this product was actually changed to "Azure Cognitive Search" in 2019.

db-engines (Andlinger): 

thank you for contacting us again!

We were struggling a little bit whether we should consider Databricks a DBMS and add it on DB-Engines, but yes, it’s there now. It will also appear in the DB-Engines Ranking from May onwards.

[omitted]

Gfesser:

Good news! I appreciate your working through this decision, and deciding to include it in your rankings henceforth. I'll inform the community, as well as my colleagues.

 

[omitted]

---

---

VP Analyst at Gartner:

It's April... and that means it's time for a fresh bowl of DBMS spaghetti. This year's DBMS revenue rankings shows cloud continuing to surge as hyperscalers solidify their market position. As always, pure-play cloud vendors are in light blue.

Database Infrastructure Competitive Director at Oracle:

Do you identify which databases are included in the market share? For example Oracle is a lot more than just Oracle Database ie: MySQL and AWS has like 15 different database services where a majority are not AWS databases?

Gfesser: Right, Gartner typically ranks vendors, not products, although methodologies are always explained at the bottom of their reports (including the criteria that vendors need to meet to begin with in order to be considered as part of a given report). So these reports are a good start, but not a be-all-end-all way to understand a given market. For database products in particular, I've relied on db-engines to provide a low bias view of the database market, although I'm curiously seeing now that they currently include Snowflake but not Databricks, so I'll be reaching out to them. I last reached out to them a few years ago, and they've always provided me prompt and succinct responses. 

Media Query Source: Part 39

---

My full responses ended up being transformed into an article at Enable Architect (June 14, 2022), after working through several rounds of editing with the journalist who initiated the media query. Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The query responses I provided to a media outlet on April 7, 2022:

Media: Career path for systems architects

Media: Systems or technology architecture is a bridge between IT operations and the c-suite. However, everyones path is different. How many years did it take you to become a systems (enterprise, solutions, data, software, technical, domain, etc.) architect or engineer?

Gfesser: I've learned that the term "architect" can mean many different things depending on the audience. The situation has gotten to the point that a fair number of firms even refuse to use this term due to some of the audience assumptions that are often made when hearing references to it. And adding qualifiers to "architect" often doesn't help much. 

While every company is arguably now a software company, many of the terms being used in industry simply aren't standardized. As such, job descriptions should be explicit as to what is expected of these roles, and the individuals writing these job descriptions should be close to the work being performed to help ensure accuracy.

 

Now, because of this situation, it's challenging to make broad statements to describe a path to become an architect. However, I've worked as the following types of architects in my career, in roughly the following order, and can explain the general process which lead to each role: software, data, enterprise, principal, and chief.

 

My technology background initially focused on being a "software engineer", now often called "software developer". After a few years, I grew into senior software engineer roles. At the time, I ran across very few individuals with the role "architect", and those who bore this title essentially put together artifacts describing the software to be built. 

 

This was before the agile movement took off, so architecture was largely seen as a step that needed to take place before development could commence. The very few times I received handoffs from architects, I found myself eventually deviating from what they had initially outlined due to these being incomplete or impractical, or because requirements changed after these were put together.

 

As the open source movement also gained momentum, I also found myself in a situation in which libraries and frameworks started to heavily influence software architecture. And because the traditional architects with whom I had periodically worked were no longer hands-on, a disconnect took place. In essence, they were no longer able to design from a practical standpoint, and I took on this responsibility as a senior software engineer.

 

Essentially, I became both senior software engineer and architect (which additionally included DevOps tasks), because the need existed to do so: separating these roles would just result in a disconnect, and working in an agile manner required that the complete solution be understood. 

 

All of this said, after working in this dual role for a few years I noticed that not only did the need to focus on data become more pronounced, but many traditional software engineers did not seem to value data because they either took it as a given as being available, thought of it as being "the easy part", did not understand it, or did not wish to work on it. 

 

It was around this time when I started my path to become a data architect as well, because I saw the work software engineers were not willing to do, and used this situation as an opportunity to add these skills to my toolbox. 

 

Alongside my software development and architecture efforts, I increasingly took on more and more data focused work. First by designing application databases and performance tuning existing databases, followed by designing and building data-intensive applications and designing and building analytical databases, leading up to my eventual decision several years ago to focus on building data-intensive products after being recruited to do so by a former manager of mine.

 

My turn in this direction came at the heels of my leading a portion of a digital transformation effort for a large client of mine, which advantageously came about after being brought on board by the CIO to work as an enterprise architect for their chief architect, who soon after asked me to take on the work of a recently departed director of architecture. For what it's worth, my team and I also built dozens of microservices while I served in this role.

 

Looking back in retrospect, it's clear that all of these roles are interrelated, speaking to my roles of enterprise architect, principal architect, and chief architect. My first experience as an enterprise architect was actually as "enterprise data architect", several years after I started working as a data architect. The distinction between data architect and enterprise data architect is the scope of the data, with enterprise involving multiple organizational units or departments, and as such, greater breadth of responsibilities, which in this case included my being asked to interview dozens of enterprise data architect candidates to build out my client's capabilities. 

 

In contrast, the roles I've since served as principal architect and chief architect have been associated with the teams of architects I've lead. For example, as chief architect I currently lead a team of "solution architects", who in turn lead a team of software architects. As it so happens, I've also spent time managing vendors and building a data engineering team while serving in this role, while also directing efforts to re-architect and open up an existing data lake, to use code for data pipelines, and to scale execution of a data-intensive product to several hundred Spark jobs during peak hour load.

 

My advice for senior software engineers looking to get into architecture is simple: be flexible with the type of work you're willing to do, and look for opportunities to branch out into new directions, realizing in the end that technology work is all interrelated, and the more practical experience you get in each area the more valuable you will become - perhaps not by your current manager, but by your next one.

---

See all of my responses to media queries here.

Media Query Source: Part 38

---

My responses ended up being included in an article at InformationWeek (May 10, 2022). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The query responses I provided to a media outlet on March 29, 2022:

Media: The good and the bad about low code.

Media: Looking for experts to explain the good, bad and the ugly about anything related to using low code development platforms. Let's make this article real-world, folks!

Gfesser: As a rule of thumb, firms likely can realistically expect some of the better low code development platforms to get them 80% of the way in terms of what they're looking to do, but it's the remaining 20% where the challenges reside. In other words, while it might be accurate to say that much of what a firm is looking to build in the near-term will be enabled by one of the better platforms, it's also likely that considerably less than 100% of mid- to long-term needs will be enabled.

From my experience, custom code will need to be eventually developed due to a variety of factors, such as unsupported features, unhandled edge cases, limited integration or limited scaling. Several years ago, for example, a health insurer CIO who had initially brought me on board as an enterprise architect for a digital transformation effort (the chief architect of whom later requested that I take over a portion of this effort following the director of architecture's departure), was sold on implementing one of the most popular low-code development platforms before the effort even started, and it didn't take long for issues to materialize.

Their presumption was that this particular platform would provide a significant jump-start with respect to building the user interface. And while this ended up being a generally accurate presumption, what wasn't taken into account was everything else that needed to be developed for the application. Quite simply, the chosen low-code development platform was built in a way which presumed use of its own components for services on which the user interface would depend. Yes, an effort to work with the low-code vendor ensued, but the jump-start didn't get the firm very far. 

In order to tackle that 20%, development teams needed to work with the platform vendor because everything was proprietary, not based on open source libraries or frameworks. Eventually, the platform was enabled to integrate with all of the custom microservices we were building, providing some needed flexibility and scaling, but there was a high dependence on the vendor in order to make this happen. What this means is that the vendor needed to modify their product, building their own code. While the development teams did not need to write this code, the vendor did, and the code was closed source. In effect, my client was funding the effort to build the vendor platform!

And it's very important to understand that it's not just what needs to be built that needs to be taken into consideration when adopting a low-code development platform. Typically, this type of product provides no version control (or version control only using a proprietary tool, kept separate from a firm's centralized version control) providing the ability to understand how instructions for a given solution have changed over time. And skillsets for such platforms aren't transferable. I've seen what initially seem to be some clever scripting languages, but apart from the incompleteness these often provide is the available talent base. Why would a strong developer want to pick up hard skills that they would never be able to use again? 

Don't get me wrong. Low-code development platforms can provide some short-term benefits, perhaps for non-technical individuals to build solutions due to a lack of developers (after all, there is a derth of talent). And perhaps these might suffice for some internally facing needs. But when it comes time to build externally facing applications for competitive advantage, low-code development platforms largely fall short. At a minimum, firms need to recognize what they're likely getting into when looking to adopt such a platform so that they can also actively plan for the long-term. As it turns out, most applications simply have a longer lifespan than initially expected.

---

See all of my responses to media queries here.

Community Comment: Part 15

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6884136604764565505?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6884136604764565505%2C6884184734960775169%29

Director at Executive Search Firm:

Do not hire an architect if you want them to be sat writing code all day long!

Sounds simple, right?

Well, as crazy as it sounds, almost every week I speak to architects who have been hired by companies who then fail to give them the chance to actually focus on architecture.

Although I appreciate the talent market is incredibly challenging for a lot of companies, do not put round-pegs into square holes.

Even if there are technical architects, or "coding architects" who enjoy contributing to the code base (and vice versa for developers with architecture), architects and developers are not the same.

Stop wasting everyone's time and hire what you need!

Gfesser: I've learned that the term "architect" can mean many different things depending on the audience. This situation has gotten to the point that a fair number of firms even refuse to use this term due to some of the audience assumptions that are often made when hearing references to it. And adding qualifiers to "architect" often doesn't help much. While every company is arguably now a software company, many of the terms being used in industry simply aren't standardized. As such, job descriptions should be explicit as to what is expected of these roles, and the individuals writing these job descriptions should be close to the work being performed to help ensure accuracy.

Community Comment: Part 14

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6871126959674400769?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6871126959674400769%2C6873206176721506304%29

Data Warehouse Architect & Developer at Healthcare Data Consultancy: I keep seeing this again and again: databases like Snowflake, Redshift, or BigQuery that are called 'Data Warehouses' when discussed in the context of acquiring them. They are not.

What they are are 'Analytical Databases': they are tools / technology fit to store data structures in to be used in an analytical context: write once, read many, good in joins, aggregations, etc...

A 'Data warehouse' is WHAT you can decide to build using these tools: a 'subject oriented, time variant, non volatile collection of data / information for analytical decision making'. Thus, Data warehousing is about implementing data architectural patterns for analytics. When building a good 'Data Warehouse' you need many more tools then only an 'Analytical Database'

Why is this important?

We can also use an 'Analytical Database' to make a complete mess of everything, to build something that definitely is NOT a "data warehouse".

I really believe we should stop naming technology as something we can BUILD by USING that technology.

After all, we also don't call a HAMMER a CHAIR, right? Using this as an example, it's immediately clear how weird this would be...

We could say: 'it is only language', who cares?

But IMHO, when working in this field of data / analytics, if there is one thing that we should try to do is use our language in a consistent way.
If not, we're really making it more complex then it needs to be...

Gfesser: Good post. Another way to put this would be to simply differentiate between infrastructure and solution. In working with some talented infrastructure folks over the years, I've often commented that with respect to data-intensive solutions, infrastructure isn't synonymous with data architecture. Data architecture is the implemented solution (models, code etc) making use of this infrastructure. Data models and data distribution strategies for Amazon Redshift, for example, aren't things that come out of the box with a Redshift deployment. Why? Because the data and use cases need to be taken into account.

Co-Founder at Data Engineering Firm: Well said. What is your take about 'data mesh'? Can I buy a solution for that?

Data Warehouse Architect & Developer at Healthcare Data Consultancy: Hi [Co-Founder at Data Engineering Firm], I would say the answer to this is a hard 'NO'. Data Mesh is such an emerging concept that it is still somehow unclear WHAT is exactly is and even more, HOW to practically implement it. What is describes is a vision of a fully decentralized data / analytics architectural pattern. A pattern where Data 'sets' are threated 'as a product'. Where ownership of data products are pushed down to 'the business'. The main work we need to do to get there has to do with culture change....

So, this example of a 'data mesh product' can, - by defacto -, NOT be fully true: https://www.starburst.io/

Senior Cloud Architect Data & AI at Microsoft: [Data Warehouse Architect & Developer at Healthcare Data Consultancy] lets agree to disagreement.

Data Warehouse Architect & Developer: ok, let me rephrase my answer: you CAN 'buy' different tools, solution components and write software that can be used to build a data mesh with. A data mesh itself, just like a data warehouse is not something that can be bought. Furthermore, there is no SINGLE technology / tool in existence that can be used to build a data mesh with.

Senior Cloud Architect Data & AI at Microsoft: [Data Warehouse Architect & Developer at Healthcare Data Consultancy] agree!

Co-Founder at Data Engineering Firm: [Data Warehouse Architect & Developer at Healthcare Data Consultancy] I think the same way. Data Mesh is not something that came out of the box. Neither the data warehouse, data lake, Lakehouse. But some companies are taking advantage of the new buzzwords in the space. I like the data mesh concept. I like Starburst. But they are not the same.

Gfesser: My team and I built a hybrid of a third and fourth generation data platform in 2019-2020, with data mesh being the fourth generation. As some other commenters have said, it takes multiple products/services and custom code to create a data mesh. A shrink-wrapped data mesh cannot be purchased. And if and when such a product could actually be purchased down the road, by that time we will already be well on our way toward a fifth generation. For anyone interested, I wrote a blog post on this topic:
Building a Data Platform on AWS from Scratch: Part 2
https://www.linkedin.com/pulse/building-data-platform-aws-from-scratch-part-2-erik-gfesser

Community Comment: Part 13

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6887723730819158016?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6887723730819158016%2C6889881950715199488%29

Programming Jokes / IT Humor & Memes Group:

python or java ??

Gfesser: Just because both Java and Python are general purpose languages doesn't mean there's a battle between them. In my opinion, there's no direct competition between these languages. I'm actually missing the humor here. Perhaps that's because I'm German, but I think stereotyping has something to do with it as well.

Media Query Source: Part 33

---

My responses ended up being included in an article at TechTarget (May 7, 2021). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The responses I provided to an "anonymous" media outlet series of queries on March 15, 2021:

Media: This article will compare Hadoop and Spark and detail how the two technologies differ and the uses for which each is suited, as well as how they aren't mutually exclusive and can be used together in some big data applications.

 

Media: What do you think are Hadoop's key strengths and weaknesses? What do you see as Sparks key strengths and weaknesses?

 

Gfesser: Before discussing the strengths and weaknesses of Apache Hadoop and Apache Spark, it helps to understand that there may exist overlap between these two in the minds of some, that Hadoop itself is often discussed in terms of its components, and that Spark was traditionally part of what became known as the Hadoop ecosystem. However, in the minds of many Hadoop is synonymous with Hadoop MapReduce. This framework made the batch processing of vast amounts of data possible by splitting it up and processing each chunk across distributed cluster nodes. At the same time, processing tends to be slow and is limited to batch, and it can be challenging to manage.

From a high level perspective, Spark can also process vast amounts of data by splitting it up and processing across distributed cluster nodes, but it typically does so much more quickly because its processing is performed in memory. And because Spark processes data in memory, it can satisfy use cases that Hadoop cannot, such as data streaming. Somewhat confusing is a component called Hadoop Streaming, which does not provide data streaming, but instead provides the ability to write MapReduce jobs in languages other than Java. Spark also supports multiple languages, but in a consistent manner that Hadoop does not provide, using APIs which resolve to common underlying code.

 

Media: How would you say Hadoop and Spark compare and contrast in terms of (please feel free to pick 3 of these): architecture, data processing capabilities, performance, scalability, applications and use cases, deployment and processing costs.

 

Gfesser: Performance: On average, benchmarks have demonstrated that Spark executes at a rate up to 100x faster than Hadoop. Additionally, the Databricks implementation of Spark is highly optimized, providing performance gains of as much as 50x that of Apache Spark. Unlike Hadoop MapReduce, Spark makes use of an advanced directed acyclic graph (DAG) processing engine which optimizes the execution plan of the processing that needs to be performed across all steps of the code that has been written to be executed by it. Combined with Spark’s use of an advanced DAG, its ability to perform processing in memory is what helps enable this speed.

Applications and use cases: While the use cases to use Hadoop and Spark overlap for batch, this isn’t the case for others. For example, Spark enables streaming of data for use cases in which decisions need to be made quickly. Some refer to this type of processing as “real-time”, but since there are so many interpretations as to what this means I typically don’t use this term, instead referring to this type of processing as non-scheduled and non-batch, subsequently exploring what is really needed in terms of timing. While other frameworks also offer the ability to stream data, some faster than Spark, Spark enables doing so for other use cases using a common framework.

 

---

See all of my responses to media queries here.

Media Query Source: Part 32

---

My responses ended up being included in an article at TechTarget (February 12, 2021) entitled "Do you need a data warehouse for business intelligence?". Above image from cited article.

---

(No transcript available for the verbal responses I provided to an "anonymous" media outlet series of queries on January 19, 2021.)

Media: Enterprises are moving further into cloud, though many organizations still have data centers. What role is the data center playing in BI today? How has that changed and how is it changing? What should organizations consider regarding their infrastructure architecture when it comes to BI? How do you know you have "the right" architecture at any point in time?  

Note that this article marks the third time I've been quoted as the result of a phone call rather than an email exchange, and unfortunately, it is also the third time I've been misquoted. So I'm 3 for 3. I'm now understanding why it's somewhat common for public figures to complain about being quoted incorrectly or out of context, and as such I'm likely to minimize future verbal exchanges with media outlets when recording is not an available option.

 

The following are what I view as misrepresentations of what I stated during my interview, and my point-by-point rebuttal:

 

Media: SPR Consulting, a digital transformation company, has a client that insisted on keeping everything in a private cloud, but it cost the company millions of dollars to build out everything they thought they might need.

 

Gfesser:

• The cited client was a past client of mine, and not necessarily a recent client (I've been in consulting most of my career).
• The cited client did not insist on keeping everything in a private data center, and the private data center they used should simply be referred to as "on premises", not a "cloud".
• The point here is that it cost the cited client millions of dollars upfront.

One key advantage of public cloud services usage is future scalability. While it needs to be determined whether chosen services will provide needed scalability, infrastructure to meet future needs doesn't need to be purchased at the outset. In other words, expenditures for unused capacity largely go away.

Media: "It's interesting how some clients will use cost as an excuse [not to move to public cloud], but if you really look at it you pay more," said Erik Gfesser, principal architect at SPR. Organizations look at how public cloud fees are charged, but they don't always consider all the factors, such as how much AWS costs per minute, Gfesser said.

Gfesser:

• The point here is that total costs should be fleshed out, regardless of which path is taken.

It's possible that use of public cloud services results in reduced costs, keeping in mind that while everything is possible, not everything is necessarily probable.

 

One example provided during my interview discussed costs alongside the AWS "shared responsibility model" which states that security of the cloud is the responsibility of AWS, whereas security in the cloud is the responsibility of the customer. Customer evaluation can overlook total costs, because costs for a given public cloud service are typically bundled with non-runtime costs, even if charges are calculated on a per-minute basis only when a given service is running.

 

---

See all of my responses to media queries here.

Community Comment: Part 7

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6739765048434135041?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6739765048434135041%2C6746484892428754944%29

Managing Director at Educational Institution: Companies are trying to make the decision between building their next-generation pub/sub systems on Pulsar or Kafka. I’ve written a post pulling my real-world experience helping companies make these decisions.
Comparing Pulsar and Kafka From a CTO’s Point of View: How a CTO would make the decision between Kafka and Pulsar
https://dzone.com/articles/comparing-pulsar-and-kafka-from-a-ctos-point-of-vi

Gfesser: Nice write-up. A brief comment on your first paragraph: "When upper management assesses a new technology, they view it from a different perspective than middle management, architects, or data engineers. Upper management is not just looking at benchmarks and feature lists; they’re looking for long-term viability and how it gives their company a clear competitive edge. In addition, they’re optimizing for time to market and costs." Architects should never evaluate options in a vacuum: evaluating near-term and long-term viability in relationship to other factors such as time-to-market and cost is typically the responsibility of the architect.

Mentoring a Data Engineering Team while Running

---

A top 10 (US) grocer client contacted SPR to help them with their data lake. While we typically implement such solutions from scratch for our clients, the non-US division of this client had already put a solution in place, and the US division with which we worked was dependent on this same infrastructure.

In summary, the company was looking for the following from us:

• Recommendations
• Mentorship & training
• Build-out of the first iterations of their data lake

One significant constraint was their need to make use of existing CDH 5.15.x (Cloudera Distribution Including Apache Hadoop) clusters already in use by the non-US division.

Additionally, not only did US data need to be kept separate from non-US data, data pipelines and DevOps pipelines also needed to be kept separate, with little visibility provided across the two. To help enable cross-team collaboration, we also needed to largely remain within the subset of Hadoop ecosystem components being used by the non-US team, with recommendations of alternative tooling, technologies, and processes kept as separate sets of deliverables.

While our client had communicated interest in recommendations, mentorship, and training, they also expressed the short term tactical need to start building out their data lake. We recommended that since the data engineering team needed working code, it made sense for us to choose use cases that addressed real business needs, tackling both at the same time.

Background on the Hadoop ecosystem

One initial challenge we had was to help ensure stakeholders understood what it means when someone refers to "Hadoop", as our initial client discovery work quickly found that management use of this term was hindering communication with respect to the work that needed to be performed.

As explained by the Apache Hadoop project, Hadoop is an open source framework that permits distributed processing of large data sets across clusters of servers using simple programming models, designed to scale up from single servers to thousands of machines, each offering local computation and storage, and to detect and handle failures at the application layer rather than relying on hardware to deliver high availability. However, notice all of the related components. These components are deployed to the same machines used by Hadoop, working together to form what is commonly referred to as the Hadoop "ecosystem".

As part of our walking through the Hadoop ecosystem with our client, we broke it down into the components already being used by the non-US team to help ensure that everyone would be on the same page as to the steps involved when someone refers to "bringing data into Hadoop".

The nearby diagram below shows a very high level representation of what this looks like from the context of a use case involving ingestion of data from a source database, and consumption of data by visualization / reporting. The large light gray box is intended to represent a single cluster node, but since a cluster contains multiple nodes, and components are distributed across all of these, with a single node not being used for all components in every case, this is a very simplified representation to show how components relate to each other with respect to integration.

We also stressed the importance of paying attention to component versions, because the developer community tends to generically reference components, and component versions can significantly affect feature availability and maturity.

While it is accurate to say that commercial distributions of Hadoop help manage version compatibility across components, this management does not negate the need for development teams to understand the capabilities of each. In the nearby table below are listed several of these components. The versions of Apache Spark and Apache Hive already being used, for example, were particularly important. 

Note that the component versions listed here are in many cases not identical to their open source equivalents, even though version numbers are the same, as Cloudera tends to bundle select bug fixes from later releases but leave the version number static save a CDH specific extension that is not included in this table.

Component	Open Source Version
	CDH 5.15.x	CDH 6.3.x	Latest
Apache Hadoop (includes HDFS, MapReduce etc)	2.6.0	3.0.0	3.3.0
Apache Hive	1.1.0	2.1.1	3.1.2
Apache Impala	2.12.0	3.2.0	3.4.0
Apache Oozie	4.1.0	5.1.0	5.2.0
Apache Parquet	1.5.0	1.9.0	1.11.1
Apache Spark	1.6.0 (2.4.0)	2.4.0	3.0.1
Apache Sqoop	1.4.6	1.4.7	1.4.7

We initially presumed use of Apache Spark 1.6.0 at the start of this engagement due to their CDH 5.15.x installation. This version of Spark is particularly old, first released in January 2016, but fortunately Apache Spark 2.4.0 had been installed by our client due to availability of the Cloudera support they desired (the reason this version is listed in parentheses). While this newer version is still 2 years old, it provides greater features and maturity that will enable eventual use of other libraries and frameworks which depend on it, such as open source Delta Lake. 

Prior to the start of this engagement, our client had planned to move to CDH 6.3.x, bringing some level of improvement, but as can be seen in the nearby table, some components have evolved faster than others, with some such as Apache Sqoop exhibiting an especially high level of inactivity in terms of community releases.

As we have advised our other clients, one sign that indicates whether a given open source component or library is a good candidate for adoption is the amount of developer community activity around it. Since many firms have moved away from Hadoop, instead focusing on Spark, other open source components not intended for deployment to Hadoop clusters have since been made available to make up for this gap. The low activity around components such as Sqoop and Oozie, combined with the high number of bugs we needed to work around during this engagement, points to industry moving elsewhere.

Data lake build-out, mentorship & training

The bulk of our time on this engagement was spent building out the first iterations of our client's data lake. As mentioned previously, the work to be done here was communicated as a short term tactical need, but as we were also tasked to enable the data engineering team, we recommended that the build-out be used as a catalyst to do so, using a real business use case for the implementation. In working with our client, we decided to implement a use case for their logistics department. 

The first step to implement this use case was to get familiar with what was needed. For example, we reviewed all of the SQL queries that were being executed against source databases, as well as the data itself. In parallel with this step, we also implemented needed HDFS folder structures and Hive Metastore databases along with needed security / permissions. We then built and tested data pipelines to ingest all of the applicable source data into what our client referred to as the "landing", "discovery", and "shared" stages (or layers) of data in the data lake. A sidebar of an earlier case study explains commonly used data stages in industry.

As explained in the key current state observations and recommendations below, we communicated our view that the name of this third stage of data was a misnomer because it was actually related to denormalized data rather than security, but we continued use of these names to help ensure collaboration between the US and non-US teams. We then walked the development team through all of the steps related to denormalization, focusing on current and potential future needs.

As we explained to our client, this third stage of data is where use case solutions cease to be generic, instead focusing on the data specific to each business domain. We demonstrated a step-by-step approach to execute this process, using pertinent logistics department data.

After getting familiar with current and potential future data needs, we rewrote current state exploratory SQL Server queries for Impala while also getting familiar with the source data read by these queries, and tested these with data ingested into the discovery stage, following up by designing denormalized tables where needed with a focus on performance. Along the way, we explained important practices to follow so that the data engineering team would be able to continue to evolve the data lake following our departure.

While data quality is mentioned in the key current state observations and recommendations below, one additional thing worth pointing out here is that we became aware of the state of data quality in source database tables during our execution of hundreds of exploratory queries against source databases.

We discovered, for example, that some source data records were being physically deleted, and some fields were not being consistently populated. In addition, many source database tables made use of natural keys rather than surrogate keys, and we explained the implications with respect to the data lake. We recommended both how to resolve these issues in source applications and how to address these issues during the data cleansing process.

Key current state observations & recommendations

As part of SPR deliverables to our client, an initial 33 sets of observations and recommendations were provided in the following 10 key areas.

Low-level:

1. code
2. data stages (layers)
3. accuracy
4. quality
5. performance

High-level:

1. infrastructure
2. security / permissions
3. development process
4. governance
5. documentation

As the US division of our client had not yet performed any development prior to our arrival, observations and recommendations were largely made for what the non-US division had already built out, although some high-level areas not specific to build-out (such as development process) also pertained to the US division.

Low-level

While the above lists are generally not ordered in terms of importance or frequency of occurrence, code is intentionally placed in the first position here because it overlaps with so many other areas, much like the case with infrastructure.

As we have mentioned to our other clients, artifacts should be versioned via code in source control. Everything should be versioned, and versioning implies code. Doing so enables consistency with deployments, the ability to track changes as a cohesive unit, and the ability of developers to independently develop and test different versions of artifacts in parallel with each other.

While exploring both formal data lake environments and version control, we noticed that some deployed object definitions, visualization / report definitions, and data pipeline code did not exist in version control. We also saw inconsistent use of formatting in the code, oftentimes making it challenging to read, and so additionally recommended artifact consistency.

For some artifact types, we relayed recommendations specifically based on community documentation. For example, since Impala stores SQL statements in lowercase, it makes sense to write this code using lowercase, so we included this step as part of the tuning process (recommendations for SQL Server code were also provided).

We provided the most observations and recommendations around the stages they made use of in their data lake. While it appeared that our client had initially followed some level of consistency, in working through what was deployed it was often challenging to determine where data should be obtained given a particular need. For example, views in lower stages of data sometimes referenced higher stages, and the presumption was that when views in one stage reference tables in another stage, the direction of these references should only be down, not up.

Denormalized tables and aggregates existed in the discovery stage rather than the shared stage, and the shared stage itself seemed misnamed, seemingly implying that it was related to security rather than denormalization. Additionally, we recommended that the landing stage for raw data solely make use of string data type and partitions for load date only, excluding attempts at use case specific performance improvement. As with traditional software development, we advised our client to resist the urge to preemptively optimize.

With respect to accuracy and quality, we noticed considerable amounts of on-the-fly data cleansing taking place in the code. As such, we recommended preemptively cleansing data in some cases in order to increase performance and decrease complexity of queries. Additionally, we noticed inappropriate data types being used in downstream data stages. For example, some data engineers made use of the double data type for numeric fields used in Impala aggregates. Because doing so can result in inconsistent aggregate calculations for visualizations / reports, we relayed the Impala project recommendation to instead make use of the decimal data type.

We also provided a high number of observations and recommendations around performance. Our first recommendation was to explicitly gather statistics following data loads to Impala, and the non-US team immediately added this step to their data pipelines so that they could start taking advantage of this improvement. Gathering statistics ensures metadata accuracy for SQL queries, which can significantly affect performance, so doing so is one of the first tuning steps that should be followed, as is the case with traditional database products.

As mentioned earlier, we recommended consistent use of the string data type in the landing stage. Apart from this stage being used for raw data, use of string data type here also enables better performance for writes just like use of parquet data type enables better performance for reads. We also noticed that some partition definitions might significantly exceed recommended Impala limits, so we relayed documentation for this aspect as well, noting that the cardinality of stored data values is used during the partition creation process.

High-level

From a high-level perspective, we provided the most observations and recommendations around infrastructure. For example, our client was making use of on premises, fixed count CDH clusters. We recommended that the firm consider use of the cloud for their Hadoop ecosystem, which would enable scalability and on-demand creation, as well as separate clusters for US and non-US divisions since cross-team data access was not being permitted anyway.

That said, Hadoop has largely fallen by the wayside in recent years, at least in terms of the batch processing provided by MapReduce, in favor of a focus on in-memory Spark and other technologies typically decoupled from what has increasingly been viewed as a heavyweight, challenging-to-manage ecosystem. We have migrated analytic workloads away from Hadoop in the past, and a preceding focus on Spark greatly enables such moves because the code becomes largely portable, enabling standalone use of Spark, although the challenge then shifts to replacing the greater ecosystem still tied to Hadoop, and ensuring that data engineers have the skills to write Scala or Python code for Spark.

A related recommendation we provided was to additionally make use of a sandbox environment that can be used for developer experimentation to help ensure stability for the separate dev environment that was serving a dual purpose as a sandbox.

During our last weeks of this engagement, we also discovered that an upgrade to CDH 6.3.3 was being performed for the first time to the dev / test cluster, rather than first being tested out in a separate upstream environment to help ensure successful upgrades of formal environments. We explained that the difference between a sandbox and dev environment is that the latter should not contain any manually deployed artifacts. Developer testing should be performed in a stable environment prior to being deployed to the test environment, and reaching a state of stability implies that a duplicate environment can be created solely from what is stored in version control.

The nearby diagram on the left below illustrates the contention that can exist, with (1) the data engineering team looking to deploy to the dev environment, (2) the infrastructure team looking to upgrade the cluster for this same environment, and (3) GitLab deploying to the test environment residing on this same cluster. All of these activities make for both an unstable environment, as well as potential delays.

Turning focus to the right, best practices instead permit the spinning up and tearing down of on-demand, standalone environments for deployment and testing that does not affect other activity, and all formally managed environments involve no manual deployment, ensuring consistency. Lastly, the activities of the infrastructure team are absorbed by the data engineering team, resulting in the intended merger of development and operations for DevOps.

We also recommended that syncing time across cluster nodes and edge nodes be implemented, as it can be challenging to follow logs for some components, as well as enabling the Python interpreter for Apache Zeppelin. While we understood that our client largely standardized on Scala and R, we also expect that many new developers will be looking to use Python since it is currently the most used data science and data engineering language. Partially because our client installed Spark 2.4.0 and no longer made use of Spark 1.6.0, we also recommended considering use of Python 3 because Python 2 is a deprecated language version no longer supported by the Python Software Foundation, even though Spark 2.4.0 supports both.

An important infrastructure piece is security / permissions, around which we presented several high-level observations and recommendations. For example, we noticed that piecemeal, manual changes were being performed, and we recommended that our client instead incorporate these as part of DevOps to ensure a repeatable process, as well as the visibility which comes along with it when everything is included as code in version control. We also recommended that local laptop development environment restrictions be lifted for developers to ensure access is available for any needed development resources, such as Maven repositories for Scala builds, in a timely manner.

Lastly, we presented observations and recommendations around development process, governance, and documentation. Based on what we saw during our engagement, we recommended itemizing what is needed to onboard new developers over time to help ensure that they can get up to speed in a timely manner, adopting a lightweight process framework such as Scrum or other, and releasing on a fixed, relatively frequent schedule coinciding with each development time period.

With respect to governance, we recommended that consistency enforced version control pull request reviews continue to be followed after our departure, that some lightweight structure for architecture and development be created in lieu of an enterprise architecture team, and that efforts such as environment upgrades be coordinated across teams, pointing back to our earlier recommendation about use of a sandbox environment.

The Result

Formal deliverables to our client included development and deployment of a new data lake for both a logistics use case, and other minor marketing use cases, including all associated data, data pipelines, DevOps pipelines, and documentation of what was implemented, as well as observations and recommendations, tuning and denormalization guidance, Git guidance, and long-term strategic guidance. 

Technologies used during this effort included CDH (Cloudera Distribution Including Apache Hadoop), Apache Hive, Apache Impala, Apache Oozie, Apache Parquet, Apache Spark, Apache Sqoop, Microsoft SQL Server, Scala, SQL, Linux Shell, Eclipse, Maven, Bitbucket (Git), Visual Studio Code. Our client was in the midst of migrating from Jenkins to GitLab during our engagement with them.

New Book Review: "The Software Architect Elevator"

New book review for The Software Architect Elevator: Redefining the Architect's Role in the Digital Enterprise, by Gregor Hohpe, O'Reilly, 2020, reposted here:

Copy provided by Amazon.

While the title of this book makes use of the term "software architect", connotations will likely arise for many potential readers. Hohpe starts discussing the term "architect" at the outset, but it isn't until the end of chapter 4 ("Enterprise Architect or Architect in the Enterprise?") where he makes an attempt to simplify what he means with respect to this book, following his discourse on "enterprise architect(ure)". "My definition of EA also implies that some IT architects, who aren't enterprise architects, work at enterprise scope. These are largely the folks I refer to in this book. Because they are the technical folks who have learned to ride the elevator (chapter 1) to the upper floors to engage with management and business architecture, they are a critical element in any IT transformation."

"How is being an 'enterprise-scale architect' different from a 'normal' IT architect? First, everything is bigger. Many large enterprises are conglomerates of different business units and divisions, each of which can be a multibillion-dollar business and can be engaged in a different business model. As things get bigger, you will also find more legacy: businesses grow over time or through acquisitions, both of which breed legacy. This legacy isn't constrained to systems, but also to people's mindsets and ways of working. Enterprise-scale architects must therefore be able to navigate organizations (chapter 34) and complex political situations. Performing true EA is as complex and as valuable as fixing a Java concurrency bug. There's enormous complexity at all levels, but the good news is that you can use similar patterns of thinking at the different levels."

"For example, software architects need to balance their system's granularity and interdependencies: a giant monolith is rather inflexible, whereas a thousand tiny services will be difficult to manage and can incur significant communication overhead. The exact same considerations apply to business architecture when considering the size of divisions and product lines. Lastly, EA also faces the same trade-offs when having to decide which systems should be centralized, which simplifies governance but can also stifle local flexibility. Architecture, if taken seriously, provides significant value at all levels."

While I largely agree with what the author presents here, it's interesting that he includes "senior developer" in the list defining what architects are _not_ in his introduction to part 1 ("Architects"). "Developers often feel they need to become an architect as the next step in their career (and their pay grade). However, becoming an architect and a superstar engineer are two different career paths, with neither being superior to the other. Architects tend to have a broader scope, including organizational and strategic aspects, whereas engineers tend to specialize and deliver running software. Mature IT organizations understand this and offer parallel career paths." As a longtime engineer and architect who has seen and experienced a lot, I find it immensely beneficial for architects to not only have significant development experience, but have these skills close at hand.

Startups often have "senior engineers" but not "architects". During my career, I've been told by quite a few startups that they don't need architects. When I've asked them who chooses the tooling and technologies, who builds proofs of concept and prototypes, who designs their software and systems alongside other stakeholders, and who provides governance, they often draw a blank. In exploring their responses to me, it becomes clear that either (1) their senior engineers act as architects, or (2) their software and systems are already in production, and they are in maintenance or some other mode. The challenge with the former is that senior engineers often don't see the big picture, and the challenge with the latter is that software and systems should arguably be built as products, which means continued evolution should be expected to take place over time.

This book is presented in 6 parts: part 1 ("Architects"), part 2 ("Architecture"), part 3 ("Communication"), part 4 ("Organizations"), part 5 ("Transformation"), and part 6 ("Epilogue: Architecting IT Transformation"). Despite my prior comments, I think the ongoing industry discussion on the role of architects is desperately needed. I've read and written a lot on this topic over the years, so I especially appreciate the subtitle to this book: "Redefining the Architect's Role in the Digital Enterprise". As with software and systems products, architects should also evolve. The challenge here is that there simply aren't any industry standard definitions for many terms commonly used in industry, and this text provides a great effort at attempting to sort everything out. Much like the term "strategy", which I've also read and written a lot about, the second part on architecture follows closely behind the discussion on architects. And in my view, the topics discussed in the third part are the bulk of what tends to shy away some senior engineers from identifying as architects, as I have personally seen a lot of paper pushing architects in the past. As the book progresses into the last two parts, the chapters increasingly resemble disjointed, separately publishable articles, although the author makes a good attempt at tying everything together.

Hohpe actually mentioned when this book was headed to the printer that it is admittedly an evolution of the "37 Things One Architect Knows About IT Transformation: A Chief Architect's Journey". At the time, he also stated that the term "software" snuck into the title because he edited the book to appeal to a broader audience of not just enterprise architects, but software architects looking for their next career step. All of this said, this work helps fill a large gap that has existed for many years, and I highly recommend it. While much of what this book presents will be common knowledge to longtime architects, especially those who have worked as consultants in both senior engineer and other leadership roles, what has largely been tribal knowledge is now arguably under one roof. Of course, not to be forgotten is that Hohpe brings his personal experience along for the ride, and as with any great book of this genre, execution eats strategy for lunch.

Community Comment: Part 4

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6727853891293089793?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6727853891293089793%2C6733693477084835840%29

Head of Artificial Intelligence at German consulting firm: This is how we have been building streaming data platforms five years ago. I'd like to have a discussion: how would you build such a thing in 2021? How would you do it in the cloud? How on-prem? If you feel like it, you make like your favorite setup as a comment.
24/7 Spark Streaming on YARN in Production
https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/


Gfesser: Spark Streaming has since been deprecated by Structured Streaming. That said, I noticed that some of the responses here reference specific commercial products, and it's good practice to look under the covers to determine which, if any, open source libraries and frameworks are being used in order to aid product comparisons and determine architectural flexibility. Some might be interested in knowing that Andreessen Horowitz very recently put together a reference architecture called "unified architecture for data infrastructure" based on input from practitioners of leading data organizations that discusses "(a) what their internal technology stacks looked like, and (b) whether it would differ if they were to build a new one from scratch", as well as 3 common blueprints.
Emerging Architectures for Modern Data Infrastructure
https://www.linkedin.com/posts/erikgfesser_the-emerging-architectures-for-modern-data-activity-6725395085095174144-VVod

Community Comment: Part 2

The comments I provided in reaction to a community discussion thread on September 21, 2020:
https://www.linkedin.com/feed/update/urn:li:ugcPost:6712112282559758336?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A6712112282559758336%2C6712156655460601856%29&replyUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A6712112282559758336%2C6713928878362025984%29

Director of Data Analytics and Application Development: Some best practices for implementing cloud data lake:
Cloud Data Lake Best Practices
https://www.linkedin.com/pulse/cloud-data-lake-best-practices-jasmeet-singh/

Senior Solutions Architect at Databricks: Its interesting that you recommend using parquet on the data lake, but delta lake only if needed. How would you perform the merge operation on parquet between layers if you don't utilize delta lake? 

Director of Data Analytics and Application Development: That is a very good question, my understanding is it depends on the architecture. Lets say data bricks or spark is not part of your architecture you can just load parquet files into time partitioned folders and over write those files. You can then create external tables in Synapse DW to serve. Delta lake has its own benefits like time travel so in some scenarios it can compliment DW layer.

Senior Solutions Architect at Databricks: Time Travel is a nice feature of Delta Lake, but some of the more beneficial features are around the management of data with common ETL patterns. The ability to Insert, Update, Delete, and Merge with simple SQL commands on cloud object storage is an amazing value add of Delta Lake. Without Delta Lake these types of operations on object storage or in data warehouse layers would be very expensive in the cloud, no? I've seen the most successful patterns emerge using Delta Lake on cheap cloud object storage to do ETL, and using a cloud data warehouse as a serving layer for use cases that require that high SLA, and can absorb the higher cost.

Director of Data Analytics and Application Development: Yes I agree but not all organizations need to use Delta Lake as serving layer, for many MPP DW with data lake storage on back end will be sufficient, you can have all the ACID compliance, insert , merge , update in DW Layer. Please check how azure synapse is doing it, they are using azure storage not built in storage so you still get benefits of lower cost storage but do not have to maintain both delta lake and DW. "With decoupled storage and compute, when using Synapse SQL pool one can: Independently size compute power irrespective of your storage needs. Grow or shrink compute power, within a SQL pool (data warehouse), without moving data. Pause compute capacity while leaving data intact, so you only pay for storage. Resume compute capacity during operational hours" Just to clarify I am not against Delta Lake I am trying to say please evaluate if you need it and in some cases it will have additional value.

Gfesser: So for best practice #3, I'm trying to reconcile your comments here with the following statement: "Implement Delta Lake only if needed, If you already have separate data warehouse layer as part of your architecture then use that instead". At a minimum, I suggest stating that Delta Lake is recommended if Spark (the minimally compatible version) is already being used, because this is the architectural dependency. Alternatively, multiple scenarios could be laid out, but since the purpose here is to explain best practices, perhaps you could just state whether you consider use of Spark a best practice. I don't think use of a "separate data warehouse layer" explains whether Delta Lake should be implemented.

Media Query Source: Part 28

---

My responses ended up being included in an article at TechTarget (November 25, 2020). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The responses I provided to an "anonymous" media outlet series of queries on October 7, 2020:

Media: I am working on a story looking at some of the hurdles organization run into when deploying big data streaming applications.

 

Media: In what significant ways have real time streaming platforms evolved over the last couple of years for providing the proper foundation for real-time analytics initiatives?

 

Gfesser: Recent real-time streaming products largely follow the same pattern, permitting ingestion of a high volume of data and the querying of this data via varying "windows" prior to being ingested by downstream processes, but there are a considerable number of competing products in this space, with commercial variations typically borrowing either open source code under the covers, or ideas brought about by open source projects. 

 

A brief side note about "real-time" processing. The vast majority of such processing is actually "soft" real-time, typically intended to mean the opposite of traditional batch processing in which data is processed in bulk, typically on a scheduled basis. I have also developed "hard" real-time software that involved processing data within strict time constraints for activity taking place in physical space. Unlike hard real-time processing, the soft real-time processing that enterprises typically need is more graceful because one of the features it increasingly provides is the ability to replay past events. Because hard real-time processing typically reacts to physical events, losing opportunities to react is probable if not carried out in a timely manner. While soft real-time events such as actions taken by users in a web application can also require timely reaction, and advantages exist in preventing such loss, the downstream loss of an upsell opportunity is likely to be considered less detrimental than worker safety in a factory.

 

In addition to permitting ingestion of a high volume of data, the querying of this data via varying windows prior to being ingested by downstream processes, and the replaying of such events, real time streaming products also typically store data for some period of time in order that events can be revisited. The time that events can be stored varies greatly. For example, Amazon Kinesis Data Streams provides the option to store data for an extended period of time of 7 days rather than the default 24 hours, but Apache Kafka provides the option to retain data for an unlimited amount of time. However, retention of data is only one factor to consider of many, as costs to store and read data also need to be taken into account, as is choosing tooling most likely to attract technology talent.

 

In-between the extremes of batch and real-time processing is what is often referred to as mini-batch, and the community has discussed at length over the last several years what this means. Essentially, rather than processing individual events one at a time, such products process events in small batches. Because costs can sometimes be cost prohibitive for individual event processing, and timeliness of such processing may not be extremely critical, there can be benefits in making use of mini-batch. When considering batch versus real-time processing, I typically discuss in the context of scheduled versus unscheduled processing. Can processing wait? What advantages might arise when not waiting and vice versa?

 

Media: What are some of the top hurdles enterprises have to overcome when deploying and managing real time data streaming platforms? For each one, what are some of the best practices for addressing these? 

 

Gfesser: The data processed by real-time streaming products is typically viewed as a stream of events. In the case of products such as Apache Kafka and Amazon Kinesis Data Streams, streams can be processed similarly to database tables, albeit the data in such tables is defined in the context of windows based on varying criteria. For example, a use case only considering data within a certain time constraint will define such tables differently than a use case that permits consideration of straggling events. Understanding these concepts can be challenging for some traditional data practitioners. 

 

Additionally, the very concept of an "event" might take some development teams time to digest, especially with respect to how events might be defined for scenarios currently processed by legacy applications. And from the perspective of managing streams, greater capabilities and flexibility typically means more development. While the capabilities and flexibility that Amazon Kinesis Data Streams and Apache Kafka offer exceed that of Amazon Kinesis Data Firehose, for example, Firehose also requires significantly less management. 

 

Note that deployment of a real-time streaming product in a greenfield scenario will be relatively less complex. When legacy applications are involved, care must be taken to sufficiently analyze existing processes to define events that these create. When defined effectively, a real-time streaming product can serve as the backbone of an entire enterprise ecosystem, providing a single source from which to draw data, especially for cases in which microservice architectures are being built. In such cases, it is prudent to tackle product selection early because it will serve as a central component. Soon afterward, make sure ongoing build efforts address ongoing management, as some teams only test with the easy path in mind and so do not end up being equipped for the longer term that often involves more data or activity.

---

See all of my responses to media queries here.

Media Query Source: Part 27

The responses I provided to an "anonymous" media outlet series of queries on August 25, 2020:

Media: The current state of Hadoop. Is it dead? Has its death been greatly exaggerated? Is it just for enterprises? How significant were/are Cloudera's 2019 stock woes? Where does Hadoop fit into the future of data engineering?

 

Gfesser: Oftentimes, the term "Hadoop" can be a bit of a misnomer these days, as it's typically the Hadoop ecosystem that's being discussed, and the Hadoop ecosystem provides quite a few components. 

 

While the Hadoop ecosystem has seen better days, it isn't technically dead. That said, the key reason that usage of it is on the wane is because of its limitations, and quite simply, more competing options have been rolled out the last several years because of this. One such option that has grown considerably in terms of adoption and community support is Apache Spark. 

 

The Hadoop ecosystem was compelling, permitting the processing of large amounts of data that either wasn't previously possible, or potentially possible with great expense. However, for a variety of reasons, including the facts that its early components limited processing to batch (which was increasingly reputed to be slow), and the need for niche specialists to develop on it and administer it, usage is decreasing relative to other technologies.

 

Spark is a component that joined other Hadoop ecosystem components, but the community has since increasingly used it standalone with file systems initially only provided by HDFS (Hadoop Distributed File System). Spark operates on data in-memory (in contrast to MapReduce), permitting the more timely stream processing of data, as well as interactive processing by end users.

 

All of this said, it's also the case that the Hadoop ecosystem provides components with functionality not provided by Spark, for example, for orchestration involving discrete data processing steps. However, because the Hadoop ecosystem is commonly seen as being unnecessarily heavyweight, firms are increasingly willing to focus on Spark and look to either commercial or open source alternatives for functionality it doesn't provide. For example, many firms use Apache NiFi or Apache Airflow for orchestration.

 

Most firms need to process data in batch, stream, and interactive modes, and while separate, specialized components can be used for each of these, it stands to reason that since Spark offers all three, use of it can help enable decreased enterprise complexity. All technologies have tradeoffs, but for many firms the Hadoop ecosystem just doesn't offer enough compelling positive tradeoffs.

 

---

See all of my responses to media queries here.

Media Query Source: Part 26

The responses I provided to an "anonymous" media outlet series of queries on July 7, 2020:

Media: Is the pace of data and database movement to the cloud accelerating?

Gfesser: Firms are increasingly leveraging the cloud to both host and process data, using both managed and containerized services.

Decreased costs are driving much of this adoption. However, increased security and flexibility are also big draws.

 

Firms which had been reluctant to use the cloud for data due to security concerns, for example, have increasingly determined that public cloud providers provide needed security.

 

AWS states, for example, that the responsibility of their cloud is to provide security “of the cloud”, keeping in mind that customers are still responsible for security “in the cloud”.

 

This responsibility is shared between AWS and its customers: AWS is responsible for protecting the infrastructure that runs its service offerings, and customers are responsible for any configuration needed by services they choose to adopt.

 

A simple example of security “in the cloud” is users, roles, and permissions: it would be unreasonable to expect AWS to take care of this type of security since the individuals making use of services will vary from customer to customer, as will the data to be accessed.

 

Media: What issues do data managers face when deploying across multi-cloud environments? How difficult is it to integrate data from (or going to) multiple clouds?

 

Gfesser: Compatibility of chosen cloud services is one of the main challenges deploying across different cloud environments.

 

It goes without saying that data integration in general has been an ongoing challenge for many firms due to the disparate systems that are typically integrated.

 

While services provided by a particular public cloud, called “cloud native” services, can provide many conveniences, firms should expect these to work differently because of the uniqueness to each respective cloud.

 

In contrast, containerized services can provide consistency in terms of how data is processed, because the same code can be used, but upfront costs can often be higher because teams are responsible for additional legwork.

 

One way to strike a balance between these two extremes is to adopt common open source products that are either containerized or used as part of a cloud native ecosystem.

 

For example, Apache Spark is a data processing framework used under the covers in many commercial products, such as AWS Glue, Databricks Unified Analytics Platform, and a variety of commercial Hadoop distributions such as AWS EMR.

 

Because each of these cloud native services makes use of what is essentially the same open source (keeping in mind the specific versions used), code written for one can often be moved between these with little to no changes, limiting the needed heavy lifting to other aspects.

 

Over time, I expect more multi-cloud products to be offered by vendors independent of the cloud providers, alleviating some of the challenges associated with multi-cloud deployments.

---

See all of my responses to media queries here.