Erik on Software

Community Comment: Part 21 Premature optimization is root of all evilThis maxim often taken out of contextFunctional correctness is first priorityExpected workloads impact architecture

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6952267780301754368?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6952267780301754368%2C6954395623710228480%29

CEO at ELT Platform Firm:

NEVER focus on load times when you first stand up a dashboard.

Load times DO NOT MATTER!

There are two paths you can take around load times:

1) Create a data stack that offers lightning-fast load times for insights that aren't useful.

If you take this path, you've generated no value for the business, and you've over-architected a tech stack and probably paid top dollar to do so. It's like setting money on fire in your backyard.

2) Find the metrics that really matter to your business and let them load super slowly until you find the right metrics.

If a dashboard truly offers valuable insights into how to run your business, stakeholders will WAIT. They'll sit there while the queries run because the insights are actually worth it.

Lucky for you, once you find the valuable insights (even if they load slowly), there are simple technical solutions to speed up load times:
- Indexing
- Materializations
- Streaming architecture
- Etc.

But these optimizations are a waste until you've identified the insights that are actually important to the business.

Find the valuable insights -> Then think about load times.

What other techniques do you use improve load times once you've figured out what matters?

Gfesser: Many of us were exposed to the maxim "premature optimization is the root of all evil" either during our computer science coursework or early in our software development careers. This maxim is just as important (perhaps even more so) for modern data work, which has been converging with software development in recent years. Some engineers, however, take this quote out of context. Here's the full quote by Donald Knuth: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, [they] will be wise to look critically at the critical code; but only *after* that code has been identified." Functional correctness is the first priority, but any expected work loads should obviously be taken into account because these will often have architectural impact.

Personal Reading List: 3Q2022

My personal reading list for July, August, and September 2022.

Color Key: Special Notes, Completed Reading. Updated throughout the Quarter.

---

At Bat (Schnell)

---

AWS Certified Database - Specialty (DBS-C01) Certification, by Kate Gawron, Packt Publishing, 2022. Copy provided by Packt publicist Nivedita Singh.

Machine Learning Engineering in Action, by Ben Wilson, Manning, 2022. Copy provided by author Ben Wilson.

---

 At Bat (Langsam)

---

The Model Thinker: What You Need to Know to Make Data Work for You, by Scott E. Page, Basic Books, 2018.

Practical Statistics for Data Scientists: 50 Essential Concepts, by Peter Bruce and Andrew Bruce, O'Reilly, 2017. Copy provided by MarkLogic.

---

 On Deck

---

Fluent Python: Clear, Concise, and Effective Programming, by Luciano Ramalho, O'Reilly, 2015.

Interpretable Machine Learning: A Guide for Making Black Box Models Explainable (Second Edition), by Christoph Molnar, Independently Published, 2022.

---

In the Hole

---

97 Things Every Data Engineer Should Know, edited by Tobias Macey, O'Reilly, 2021. Copy provided by Promethium.

Presto: The Definitive Guide, by Matt Fuller, Manfred Moser, and Martin Traverso, O'Reilly, 2020. Copy provided by Starburst.