Erik on Software

Media Query Source: Part 40

---

My responses ended up being included in an article at CIO (June 7, 2022). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The query responses I provided to a media outlet on May 6, 2022:

Media: I'm contributing to an article on 10 key roles every AI team needs. What are the key titles, what does each function do? (For example, data scientists, ML engineers, product managers, data engineers.) I'd love to hear from individuals holding those jobs who can briefly talk about what their roles entail, or enterprise AI managers who manage these teams.

Gfesser: I'll speak to "data engineer", a role I've served in the past and a role for which I currently hire and manage, although I have served in other related roles as well.

While the roles of data engineer and ML engineer are closely related, I consider the data engineer foundational for both ML and non-ML initiatives.

In a nutshell, data engineers build data pipelines to collect and assemble data for downstream usage, and in a DevOps world, additionally build pipelines to implement the infrastructure on which these data pipelines run.

For example, when implementing data pipelines in one of the public clouds, a data engineer needs to first write the scripts to spin up the necessary cloud services which provide the compute necessary to process ingested data.

This ingested data is then transformed and subsequently stored in increasing states of consumption readiness, which can then be consumed for a variety of uses, including ML.

Data engineers are typically required to be skilled in languages such as Python, SQL, and Scala, libraries used for cleansing and transforming data, and when large volumes of data are involved, de facto standard Apache Spark for distributed data processing.

Data engineers are considered foundational because data needs to be collected and made suitable for consumption before anything trustworthy can be done with it.

Note that the role of data engineer has evolved over the last several years to a point that, not only is it considered critical to all serious data initiatives, it is also converging with other roles needed by ML teams.

Instead of waiting for a data engineer to build what is necessary for a data consumption use case, for example, other roles such as data scientist can instead do this work themselves, or vice versa.

That said, it is typically easier for a given skilled data engineer to pick up data science skills, due to their presumed software development background and because industry has not placed enough attention on building data engineering skills relative to data science.

---

See all of my responses to media queries here.

Community Comment: Part 19

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6926625786309316608?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6926625786309316608%2C6926689203154018305%29

CEO at ELT platform firm:

Do you think your data team is special?
It’s not.
You take one of two approaches.

Strategy 1. Invest in people in service of open source software.
Strategy 2. Invest in SaaS software in service of people.
Your approach directly determines who you hire, how you spend money, and which technologies you use.

Let me outline the two approaches in more detail.

Approach 1.
Teams that invest in people in service of open source software.
This is common in a handful of situations:
- When data teams are born out of engineering teams.
- Data teams within particular industry verticals - like healthcare and financial services.
- Very large enterprises (i.e. the Fortune 500).
- Non-US companies (where SaaS software for data management is less accessible).
These teams invest heavily in headcount.
The biggest line item for the data team will be engineering talent on staff.

Approach 2.
Teams that invest in SaaS software in service of people.
This is common in a handful of scenarios:
- When data teams are built out under a business function.
- Data teams within high-growth industries, with lighter weight procurement processes (eCommerce, marketing, etc.).
- Mid-market companies with lean teams.
- US-based companies.
These teams invest heavily in SaaS software.
As a result, budgets are more evenly split between headcount (typically very lean teams) and SaaS software.

How can you tell which type of team you’re building?
There are two easy tests:
1. When you see a problem and need a solution, do you first try to build in-house? Or do you first try to find a SaaS solution?
2. Is your team primarily budgeting for headcount or software?

I only came to this realization after I started selling to data teams.
Why?
As a vendor in the data world, you start by thinking all data teams are the same.
At [ELT platform firm], we’re a SaaS solution.
We’ve realized we’re not a good fit for teams that invest in people in service of open source software.
The idea of leveraging a SaaS solution goes against the culture, the procurement methodology, and the architecture of those companies.
Have other vendors arrived at the same realization?

For those people in analytics.
Which type of data team are you building?

Gfesser: Interesting thoughts. In the context of your argument, I would typically call strategy #1 "invest in people in service of custom development" because the opposite of buying software is building software, and making use of open source tech is one of the options for construction. Tech savvy firms will obviously make heavy use of open source tech. But there is something else to consider that I often see missing in discussions such as this: buying commercialized open source. In other words, commercial software which has been built on open source. AWS and GCP, for example, provide quite a few examples of such offerings. But I don't think the quantities of talent for these two strategies are at opposite ends of a spectrum.

Community Comment: Part 18

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6931840271642951680?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6931840271642951680%2C6931916091128500224%29

Product Owner at Consultancy:

“A developer should be writing at least as much in code as I write as part of my documentation.”

Read this in a post by a PM. I guess he meant it as a joke (giving the benefit of doubt), but generally I’ve seen this attitude of judging a developer’s work by the lines of code they write.

“I saw you checked-in your code. This took two days?”, they ask. And then the developer explains why!

Whenever I see this happening, I get reminded of [someone's] post on ‘They don’t pay you to type’ where he argues that a developer’s worth cannot be judged by the lines of code they write.

I have personally seen problems that were solved with two lines of code, but that took several days for the developer to crack. I’ve also seen cases where an experienced developer solved a complicated problem within a few minutes, but pretended to take time, otherwise no one will see worth in their work. (This becomes further problematic in case of freelancers where they’re denied promised money because “it just took a few minutes/few lines”.)

Developers aren’t typists. They are problem-solvers.

You don’t deduct a sprinter’s salary because they finished hundred metres in few seconds. You reward them for the preparation that enabled them to do this.

Gfesser: Thanks for weaving in a running analogy. As a lifelong runner, I rarely see this in posts. However, as a former sprinter and middle distance runner in school, I look at this a bit differently. Unfortunately, there are no rewards for race preparation. At least, I'm not a believer in participation awards. The sprinter should be rewarded for their good finish times just like a programmer should be rewarded for their good work.

Product Owner at Consultancy:

I’m not saying anyone should necessarily reward participation. I’m saying, rewards for good work are not just for the work, they are also for the preparation. Preparation is being rewarded because of the work, not just for itself.

Gfesser: I think I hear what you're saying. In the end, though, it's results that arguably matter. Going back to the running analogy, I've come across runners who were even more naturally gifted than myself, and as such they didn't necessarily need as much preparation as me. Placing in races, however, is what mattered, not me trying harder than them. Is that fair? Yes, because finish times are objective. :-)