Erik on Software

Community Post: Part 1 My open letter to DB-EnginesGartner's "DBMS market share ranks" postDatabricks left out of DB-Engines rankingMy proposal to DB-Engines was successful

---

Background for a community post I wrote:
https://www.linkedin.com/posts/erikgfesser_gartner-databricks-database-activity-6925028958199570432-t5bW/

Gfesser: Success! About a week ago, I submitted the following comment on a "DBMS Market Share Ranks" post made by a VP at Gartner. After subsequently following up with db-engines, they have just informed me that, after careful consideration, my proposal to add Databricks to their monthly ranking has been approved, and it will be included starting in May 2022 (link to my original post in the comments below). In the meantime, Databricks has been added to their catalog of 391 database products (link to this catalog in the comments below).

New db-engines entry for Databricks:
https://db-engines.com/en/system/Databricks

---

My open letter to the folks at db-engines on April 20, 2022:

Gfesser:

I've been a heavy user of db-engines over the years, and continue to appreciate the information it provides.

I last contacted you in June 2017, and had some brief exchanges with Dr Paul Andlinger.

The other day, I visited db-engines for the first time in a while, and noticed that Databricks isn't yet represented in the rankings. 

I'll greatly appreciate your explaining why this is the case, as well as your considering it being added.

The other day, I had a conversation on LinkedIn about the latest Gartner ranking on database vendors, and I noticed that Snowflake is present in both the Gartner and db-engines rankings, but Databricks is only present in the Gartner ranking.

I told the community that I would follow up with you on this.

In recent months, I've introduced use of Databricks SQL to a data platform for which I serve as chief architect. Storage and compute are separate, with compute provided by Databricks (Spark) clusters and storage provided by the data lake. SQL queries can be executed against any data for which Hive tables have been created.

Based on db-engines database categorizations, it looks like Databricks would fit in the "relational DBMS" category. 

Other similar products I've used, such as Presto and Impala, are also represented in this category. You might also want to consider adding Amazon Athena to this category at the same time, which is built on Presto. 

Also, I happened to notice in the "search engine" category that you've included "Microsoft Azure Search": the name of this product was actually changed to "Azure Cognitive Search" in 2019.

db-engines (Andlinger): 

thank you for contacting us again!

We were struggling a little bit whether we should consider Databricks a DBMS and add it on DB-Engines, but yes, it’s there now. It will also appear in the DB-Engines Ranking from May onwards.

[omitted]

Gfesser:

Good news! I appreciate your working through this decision, and deciding to include it in your rankings henceforth. I'll inform the community, as well as my colleagues.

 

[omitted]

---

---

VP Analyst at Gartner:

It's April... and that means it's time for a fresh bowl of DBMS spaghetti. This year's DBMS revenue rankings shows cloud continuing to surge as hyperscalers solidify their market position. As always, pure-play cloud vendors are in light blue.

Database Infrastructure Competitive Director at Oracle:

Do you identify which databases are included in the market share? For example Oracle is a lot more than just Oracle Database ie: MySQL and AWS has like 15 different database services where a majority are not AWS databases?

Gfesser: Right, Gartner typically ranks vendors, not products, although methodologies are always explained at the bottom of their reports (including the criteria that vendors need to meet to begin with in order to be considered as part of a given report). So these reports are a good start, but not a be-all-end-all way to understand a given market. For database products in particular, I've relied on db-engines to provide a low bias view of the database market, although I'm curiously seeing now that they currently include Snowflake but not Databricks, so I'll be reaching out to them. I last reached out to them a few years ago, and they've always provided me prompt and succinct responses. 

Personal Reading List: 2Q2022

My personal reading list for April, May, and June 2022.

Color Key: Special Notes, Completed Reading. Updated throughout the Quarter.

---

At Bat (Schnell)

---

How Will You Measure Your Life?, by Clayton M. Christensen, James Allworth, and Karen Dillon, Harper Business, 2012.

Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (Second Edition), by Christoph Molnar, Independently Published, 2022.

Machine Learning Engineering in Action, by Ben Wilson, Manning, 2022. Copy provided by author Ben Wilson.

---

 At Bat (Langsam)

---

Fluent Python: Clear, Concise, and Effective Programming, by Luciano Ramalho, O'Reilly, 2015.

The Model Thinker: What You Need to Know to Make Data Work for You, by Scott E. Page, Basic Books, 2018.

Practical Statistics for Data Scientists: 50 Essential Concepts, by Peter Bruce and Andrew Bruce, O'Reilly, 2017. Copy provided by MarkLogic.

---

 On Deck

---

Enterprise AIOps: A Framework for Enabling Artificial Intelligence, by Justin Neroda, Steve Escaravage, and Aaron Peters, O'Reilly, 2021. Copy provided by Booz Allen Hamilton.

Work Better Together: How to Cultivate Strong Relationships to Maximize Well-Being and Boost Bottom Lines, by Jen Fisher and Anh Phillips, McGraw Hill, 2021. Copy provided by Deloitte.

Running with Purpose: How Brooks Outpaced Goliath Competitors to Lead the Pack, by Jim Weber, HarperCollins Leadership, 2022. Copy provided by Amazon.

---

In the Hole

---

AWS Certified Database - Specialty (DBS-C01) Certification, by Kate Gawron, Packt Publishing, 2022. Copy provided by Packt publicist Nivedita Singh.

Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith, by Sam Newman, O'Reilly, 2020. Copy provided by Nginx.

Presto: The Definitive Guide, by Matt Fuller, Manfred Moser, and Martin Traverso, O'Reilly, 2020. Copy provided by Starburst.

Media Query Source: Part 39 Enable Architect (US digital magazine)My career journey from engineer to architect"Architect" definition dependent on audiencePractical experience & flexibility bring value

---

My full responses ended up being transformed into an article at Enable Architect (June 14, 2022), after working through several rounds of editing with the journalist who initiated the media query. Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The query responses I provided to a media outlet on April 7, 2022:

Media: Career path for systems architects

Media: Systems or technology architecture is a bridge between IT operations and the c-suite. However, everyones path is different. How many years did it take you to become a systems (enterprise, solutions, data, software, technical, domain, etc.) architect or engineer?

Gfesser: I've learned that the term "architect" can mean many different things depending on the audience. The situation has gotten to the point that a fair number of firms even refuse to use this term due to some of the audience assumptions that are often made when hearing references to it. And adding qualifiers to "architect" often doesn't help much. 

While every company is arguably now a software company, many of the terms being used in industry simply aren't standardized. As such, job descriptions should be explicit as to what is expected of these roles, and the individuals writing these job descriptions should be close to the work being performed to help ensure accuracy.

 

Now, because of this situation, it's challenging to make broad statements to describe a path to become an architect. However, I've worked as the following types of architects in my career, in roughly the following order, and can explain the general process which lead to each role: software, data, enterprise, principal, and chief.

 

My technology background initially focused on being a "software engineer", now often called "software developer". After a few years, I grew into senior software engineer roles. At the time, I ran across very few individuals with the role "architect", and those who bore this title essentially put together artifacts describing the software to be built. 

 

This was before the agile movement took off, so architecture was largely seen as a step that needed to take place before development could commence. The very few times I received handoffs from architects, I found myself eventually deviating from what they had initially outlined due to these being incomplete or impractical, or because requirements changed after these were put together.

 

As the open source movement also gained momentum, I also found myself in a situation in which libraries and frameworks started to heavily influence software architecture. And because the traditional architects with whom I had periodically worked were no longer hands-on, a disconnect took place. In essence, they were no longer able to design from a practical standpoint, and I took on this responsibility as a senior software engineer.

 

Essentially, I became both senior software engineer and architect (which additionally included DevOps tasks), because the need existed to do so: separating these roles would just result in a disconnect, and working in an agile manner required that the complete solution be understood. 

 

All of this said, after working in this dual role for a few years I noticed that not only did the need to focus on data become more pronounced, but many traditional software engineers did not seem to value data because they either took it as a given as being available, thought of it as being "the easy part", did not understand it, or did not wish to work on it. 

 

It was around this time when I started my path to become a data architect as well, because I saw the work software engineers were not willing to do, and used this situation as an opportunity to add these skills to my toolbox. 

 

Alongside my software development and architecture efforts, I increasingly took on more and more data focused work. First by designing application databases and performance tuning existing databases, followed by designing and building data-intensive applications and designing and building analytical databases, leading up to my eventual decision several years ago to focus on building data-intensive products after being recruited to do so by a former manager of mine.

 

My turn in this direction came at the heels of my leading a portion of a digital transformation effort for a large client of mine, which advantageously came about after being brought on board by the CIO to work as an enterprise architect for their chief architect, who soon after asked me to take on the work of a recently departed director of architecture. For what it's worth, my team and I also built dozens of microservices while I served in this role.

 

Looking back in retrospect, it's clear that all of these roles are interrelated, speaking to my roles of enterprise architect, principal architect, and chief architect. My first experience as an enterprise architect was actually as "enterprise data architect", several years after I started working as a data architect. The distinction between data architect and enterprise data architect is the scope of the data, with enterprise involving multiple organizational units or departments, and as such, greater breadth of responsibilities, which in this case included my being asked to interview dozens of enterprise data architect candidates to build out my client's capabilities. 

 

In contrast, the roles I've since served as principal architect and chief architect have been associated with the teams of architects I've lead. For example, as chief architect I currently lead a team of "solution architects", who in turn lead a team of software architects. As it so happens, I've also spent time managing vendors and building a data engineering team while serving in this role, while also directing efforts to re-architect and open up an existing data lake, to use code for data pipelines, and to scale execution of a data-intensive product to several hundred Spark jobs during peak hour load.

 

My advice for senior software engineers looking to get into architecture is simple: be flexible with the type of work you're willing to do, and look for opportunities to branch out into new directions, realizing in the end that technology work is all interrelated, and the more practical experience you get in each area the more valuable you will become - perhaps not by your current manager, but by your next one.

---

See all of my responses to media queries here.