Erik on Software

Community Comment: Part 22

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6966251443859132416?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6966251443859132416%2C6966340024552353792%29

CEO at Consulting Firm:

Software meetings best practices:

❌ Never say: Never.
✅ Always say: It depends.

Gfesser: As a consultant, I used the phrase "it depends" many times with clients. One such occasion I fondly recall involved an initial meeting with a potential client in their expansive corner office overlooking the Chicago River. In the middle of the meeting, I was asked point blank, given a choice of tech stack, what I would recommend using if building out their product. I said, "well, it depends". Their response can be summarized as follows: "Consultants still use that line? Why can't you just answer the question?" I proceeded by explaining some factors that need to be taken into account in order to make that type of decision, factors he wasn't able to address during that meeting. The result? We won the job, and I led architecture and development, writing all of the initial code. The client made their first sales to customers within a year. Never bow to unjustified pressure: your clients and customers will eventually thank you.

Media Query Source: Part 39

---

My full responses ended up being transformed into an article at Enable Architect (June 14, 2022), after working through several rounds of editing with the journalist who initiated the media query. Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The query responses I provided to a media outlet on April 7, 2022:

Media: Career path for systems architects

Media: Systems or technology architecture is a bridge between IT operations and the c-suite. However, everyones path is different. How many years did it take you to become a systems (enterprise, solutions, data, software, technical, domain, etc.) architect or engineer?

Gfesser: I've learned that the term "architect" can mean many different things depending on the audience. The situation has gotten to the point that a fair number of firms even refuse to use this term due to some of the audience assumptions that are often made when hearing references to it. And adding qualifiers to "architect" often doesn't help much. 

While every company is arguably now a software company, many of the terms being used in industry simply aren't standardized. As such, job descriptions should be explicit as to what is expected of these roles, and the individuals writing these job descriptions should be close to the work being performed to help ensure accuracy.

 

Now, because of this situation, it's challenging to make broad statements to describe a path to become an architect. However, I've worked as the following types of architects in my career, in roughly the following order, and can explain the general process which lead to each role: software, data, enterprise, principal, and chief.

 

My technology background initially focused on being a "software engineer", now often called "software developer". After a few years, I grew into senior software engineer roles. At the time, I ran across very few individuals with the role "architect", and those who bore this title essentially put together artifacts describing the software to be built. 

 

This was before the agile movement took off, so architecture was largely seen as a step that needed to take place before development could commence. The very few times I received handoffs from architects, I found myself eventually deviating from what they had initially outlined due to these being incomplete or impractical, or because requirements changed after these were put together.

 

As the open source movement also gained momentum, I also found myself in a situation in which libraries and frameworks started to heavily influence software architecture. And because the traditional architects with whom I had periodically worked were no longer hands-on, a disconnect took place. In essence, they were no longer able to design from a practical standpoint, and I took on this responsibility as a senior software engineer.

 

Essentially, I became both senior software engineer and architect (which additionally included DevOps tasks), because the need existed to do so: separating these roles would just result in a disconnect, and working in an agile manner required that the complete solution be understood. 

 

All of this said, after working in this dual role for a few years I noticed that not only did the need to focus on data become more pronounced, but many traditional software engineers did not seem to value data because they either took it as a given as being available, thought of it as being "the easy part", did not understand it, or did not wish to work on it. 

 

It was around this time when I started my path to become a data architect as well, because I saw the work software engineers were not willing to do, and used this situation as an opportunity to add these skills to my toolbox. 

 

Alongside my software development and architecture efforts, I increasingly took on more and more data focused work. First by designing application databases and performance tuning existing databases, followed by designing and building data-intensive applications and designing and building analytical databases, leading up to my eventual decision several years ago to focus on building data-intensive products after being recruited to do so by a former manager of mine.

 

My turn in this direction came at the heels of my leading a portion of a digital transformation effort for a large client of mine, which advantageously came about after being brought on board by the CIO to work as an enterprise architect for their chief architect, who soon after asked me to take on the work of a recently departed director of architecture. For what it's worth, my team and I also built dozens of microservices while I served in this role.

 

Looking back in retrospect, it's clear that all of these roles are interrelated, speaking to my roles of enterprise architect, principal architect, and chief architect. My first experience as an enterprise architect was actually as "enterprise data architect", several years after I started working as a data architect. The distinction between data architect and enterprise data architect is the scope of the data, with enterprise involving multiple organizational units or departments, and as such, greater breadth of responsibilities, which in this case included my being asked to interview dozens of enterprise data architect candidates to build out my client's capabilities. 

 

In contrast, the roles I've since served as principal architect and chief architect have been associated with the teams of architects I've lead. For example, as chief architect I currently lead a team of "solution architects", who in turn lead a team of software architects. As it so happens, I've also spent time managing vendors and building a data engineering team while serving in this role, while also directing efforts to re-architect and open up an existing data lake, to use code for data pipelines, and to scale execution of a data-intensive product to several hundred Spark jobs during peak hour load.

 

My advice for senior software engineers looking to get into architecture is simple: be flexible with the type of work you're willing to do, and look for opportunities to branch out into new directions, realizing in the end that technology work is all interrelated, and the more practical experience you get in each area the more valuable you will become - perhaps not by your current manager, but by your next one.

---

See all of my responses to media queries here.

Community Comment: Part 15

The comments I provided in reaction to a community discussion thread:
https://www.linkedin.com/feed/update/urn:li:activity:6884136604764565505?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6884136604764565505%2C6884184734960775169%29

Director at Executive Search Firm:

Do not hire an architect if you want them to be sat writing code all day long!

Sounds simple, right?

Well, as crazy as it sounds, almost every week I speak to architects who have been hired by companies who then fail to give them the chance to actually focus on architecture.

Although I appreciate the talent market is incredibly challenging for a lot of companies, do not put round-pegs into square holes.

Even if there are technical architects, or "coding architects" who enjoy contributing to the code base (and vice versa for developers with architecture), architects and developers are not the same.

Stop wasting everyone's time and hire what you need!

Gfesser: I've learned that the term "architect" can mean many different things depending on the audience. This situation has gotten to the point that a fair number of firms even refuse to use this term due to some of the audience assumptions that are often made when hearing references to it. And adding qualifiers to "architect" often doesn't help much. While every company is arguably now a software company, many of the terms being used in industry simply aren't standardized. As such, job descriptions should be explicit as to what is expected of these roles, and the individuals writing these job descriptions should be close to the work being performed to help ensure accuracy.

Media Query Source: Part 25

---

My responses ended up being included in the Interview Series at Unite.AI (April 21, 2020), a San Francisco based media firm "designed to offer detailed analysis and news on the latest advancements in machine learning and AI technology", including all things data. In this case, the entire question and answer exchange between Antoine Tardif and myself was published. Above image from cited article.

---

The responses I provided to an "anonymous" media outlet series of queries on April 15, 2020:

Media: What initially attracted you to machine learning?

Gfesser: Its enablement of applications to continuously learn. I had started my development career as a senior data analyst using SPSS at what became a global market research firm, and later incorporated use of a business rules engine called Drools into applications that I built for clients, but the output for all of this work was essentially static. I later worked through process improvement training, during which time instructors demonstrated in detail how they were able to improve, through statistics and other methods, business processes used by their clients, but here again the output was largely focused on points in time. My experience working to improve a healthcare product my colleagues and I built during this same time period is what showed me why continuous learning is necessary for such efforts, but the resources now available did not exist back then. Interestingly, my attraction to machine learning has come full circle, as my graduate adviser cautioned me against a specialty in what was then called artificial intelligence, due to the AI winter at the time. I choose to instead make use terms such as ML because these hold fewer connotations, and because even AWS acknowledges that its AI services layer is really just a higher level abstraction built on top of its ML services layer. While some of the ML hype out there is unrealistic, it provides powerful capabilities from the perspective of developers, as long as these same practitioners acknowledge the fact that the value which ML provides is only as good as the data processed by it.   

Media: You’re a huge open source advocate, could you discuss why open source is so important?

Gfesser: One aspect about open source that I've needed to explain to executives over the years is that the primary benefit of open source is not that use of such software is made available without monetary cost, but that the source code is made freely available. Additionally, developers making use of this source code can modify it for their own use, and if suggested changes are approved, make these changes available to other developers using it. In fact, the movement behind open source software started due to developers waiting at length for commercial firms to make changes to products they licensed, so developers took it upon themselves to write software with the same functionality, opening it up to be improved upon by other developers. Commercialized open source takes advantage of these benefits, the reality being that many modern products make use of open source under the covers, even whilst commercial variants of such software typically provide additional components not available as part of a given open source release, providing differentiators as well as support if this is needed. My first experiences with open source took place while building the healthcare product I mentioned earlier, making use of tooling such as Apache Ant, used to build software, and an early DevOps product at the time called Hudson (the code base of which later became Jenkins). The primary reason behind our decisions to use these open source products was that these either provided better solutions to commercial alternatives, or were innovative solutions not even offered by commercial entities, not to mention that the commercial licensing of some of the products we had been using was overly restrictive, leading to excessive red tape when it came time to needing more licenses, due to the costs involved. Over time, I've seen open source offerings continue to evolve, providing much needed innovation. For example, many of the issues with which my colleagues and I wrestled building this healthcare product were later solved by an innovative open source Java product we started using called Spring Framework, which is still going strong after more than a decade, the ecosystem of which now stretches far beyond some of the innovations that it initially provided, now seen as commonplace, such as dependency injection. 

Media: You’ve used open source for the building of PoCs, prototypes, and MVPs. Could you share your journey behind some of these products?

Gfesser: As explained in one of the guiding principles I presented to a recent client, build-outs for the data platform we built for them should continue to be iteratively carried out as needed over time. The components built out for this platform should not be expected to remain static, as needs change and new components and component features will be made available over time. When building out platform functionality, always start with what is minimally viable before adding unneeded bells and whistles, which in some cases even includes configuration. Start with what is functional, make sure you understand it, and then evolve it. Don’t waste time and money building what has low likelihood of being used, but make an effort to get ahead of future needs. The MVP we built for this product expressly needed to be built so that additional use cases could continue to be built on top of it, even though it came packaged with implementation of a single use case, for expense anomaly detection. Unlike this client, an earlier product that I built had some history behind it prior to my arrival. In this case, stakeholders had been debating for three years (!) how they should approach a product they were looking to build. A client executive explained that one of the reasons he brought me in was to help the firm get past some of these internal debates, especially because the product that he was looking to build needed to satisfy the hierarchy of organizations involved. I came to find that these turf wars were largely associated with the data owned by the client, its subsidiaries, and its external customers, so in this case the entire product backlog revolved around how this data would be ingested, stored, secured, and consumed for a single use case generating on-the-fly networks of healthcare providers for cost analyses. Earlier in my career, I came to understand that an architectural quality called "usability" was not limited to just end users, but software developers themselves. The reason this is the case is because the code that is written needs to be usable just like user interfaces need to be usable by end users. In order that a product become usable, proofs of concept need to be built to demonstrate that developers are going to be able to do what they set out to do, especially when related to the specific technology choices they are making. But proofs of concept are just the beginning, as products are best when evolved over time. In my view, the foundation for an MVP, however, should ideally be built on prototypes exhibiting some stability so that developers will be able to continue to evolve it.

Media: While reviewing the book "Machine Learning at Enterprise Scale" you stated that "use of open source products, frameworks, and languages alongside an agile architecture composed of a mix of open source and commercial components provides the nimbleness that many firms need but don't immediately realize at the outset". Could you go into some details as to why you believe that firms which use open source are more nimble?

 

Many commercial data products use key open source components under the covers, and enable developers to use popular programming languages such as Python. The firms which build these products know that the open source components they've chosen to incorporate give them a jump start when these are already widely used by the community. Open source components with strong communities are easier to sell, due to the familiarity that these bring to the table. Commercially available products which consist mainly of closed source, or even open source that is largely only used by specific commercial products, often require either training by these vendors, or licenses in order to make use of the software. Additionally, documentation for such components is largely not made publicly available, forcing the continued dependency of developers on these firms. When widely accepted open source components such as Apache Spark are the central focus, as with products such as Databricks Unified Analytics Platform, many of these items are already made available in the community, minimizing the portions on which development teams need to depend on commercial entities to do their work. Additionally, because components such as Apache Spark are broadly accepted as de facto industry standard tooling, code can also be more easily migrated across commercial implementations of such products. Firms will always be inclined to incorporate what they view as competitive differentiators, but many developers don't want to use products that are completely novel because this proves challenging to move between firms, and tends to cut their ties with the strong communities they have come to expect. From personal experience, I've worked with such products in the past, and it can be challenging to get competent support. And this is ironic, given that such firms sell their products with the customer expectation that support will be provided in a timely manner. I've had the experience submitting a pull request to an open source project, with the fix incorporated into the build that same day, but cannot say the same about any commercial project with which I have worked. 

Media: Something else that you believe about open source is that it leads to "access to strong developer communities." How large are some of these communities and what makes them so effective?

 

Gfesser: Developer communities around a given open source product can reach into the hundreds of thousands. Adoption rates don't necessarily point to community strength, but are a good indicator that this is the case due to their tendency to produce virtuous cycles. I consider communities to be strong when these produce healthy discussion and effective documentation, and where active development is taking place. When an architect or senior developer works through the process to choose which such products to incorporate into what they are building, many factors typically come into play, not only about the product itself and what the community looks like, but about the development teams who will be adopting these, whether these are a good fit for the ecosystem being developed, what the roadmap looks like, and in some cases whether commercial support can be found in the case this may be needed. However, many of these aspects fall by the wayside in the absence of strong developer communities.

Media: You have reviewed 100s of books on your website, are there three that you could recommend to our readers? (with links to your reviews for these books as well)

 

Gfesser: These days I read very few programming books, and while there are exceptions, the reality is that these are typically outdated very quickly, and the developer community usually provides better alternatives via discussion forums and documentation. Many of the books I currently read are made freely available to me, either via technology newsletters to which I subscribe, authors and publicists who reach out to me, or the ones Amazon sends to me. For example, Amazon sent me a pre-publication uncorrected proof of "The Lean Startup" for my review in 2011, introducing me to the concept of the MVP, and just recently sent me a copy of "Julia for Beginners". (1) One book from O'Reilly that I've recommended is "In Search of Database Nirvana". The author covers in detail the challenges for a data query engine to support workloads spanning the spectrum of OLTP on one end, to analytics on the other end, with operational and business intelligence workloads in the middle. This book can be used as a guide to assess a database engine or combination of query and storage engines, geared toward meeting one's workload requirements, whether these be transactional, analytical, or a mix of these two. Additionally, the author's coverage of the "swinging database pendulum" in recent years is especially well done. (2) While much has changed in the data space over the last few years, since new data analytics products continue to be introduced, "Disruptive Analytics" presents an approachable, short history of the last 50 years of innovation in analytics that I haven't seen elsewhere, and discusses two types of disruption: disruptive innovation within the analytics value chain, and industry disruption by innovations in analytics. From the perspective of startups and analytics practitioners, success is enabled by disrupting their industries, because using analytics to differentiate a product is a way to create a disruptive business model or to create new markets. From the perspective of investing in analytics technology for their organizations, taking a wait-and-see approach might make sense because technologies at risk of disruption are risky investments due to abbreviated useful lifespans. (3) One of the best technology business texts I've read is "The Limits of Strategy", by a co-founder of Research Board (acquired by Gartner), an international think tank that investigates developments in the computing world and how corporations should adapt. The author presents very detailed notes from many of his conversations with business leaders, providing insightful analysis throughout about his experiences building (with his wife) a group of clients, major firms that needed to mesh their strategies with the exploding world of computing. As I commented in my review, what sets this book apart from other related efforts are two seemingly opposed characteristics: industry-wide breadth, and intimacy that is only available through face-to-face interaction.

Media: You are the Principal Architect for the data practice of SPR. Could you describe what SPR does?

Gfesser: SPR is a digital technology consultancy based in the Chicago area, delivering technology projects for a range of clients, from Fortune 1000 enterprises to local startups. We build end-to-end digital experiences using a range of technology capabilities, everything from custom software development, user experience, data, and cloud infrastructure, to DevOps coaching, software testing, and project management. 

Media: What are some of your responsibilities with SPR?

Gfesser: As principal architect, my key responsibility is to drive solution delivery for clients, leading architecture and development for projects, and this often means wearing other hats such as product owner because being able to relate to how products are built from a hands-on perspective weighs heavily in regard to how work should be prioritized, especially when building from scratch. I'm also pulled in to discussions with potential clients when my expertise is needed, and the company recently requested that I start an ongoing series of sessions with fellow architects in the data practice to discuss client projects, side projects, and what my colleagues are doing to keep abreast of technology, similar to what I had run for a prior consultancy, albeit the internal meetups so-to-speak for this other firm involved their entire technology practice, not specific to data work. For the bulk of my career, I've specialized in open source development using Java, performing an increasing amount of data work along the way. In addition to these two specializations, I also do what my colleagues and I have come to call "practical" or "pragmatic" enterprise architecture, which means performing architecture tasks in the context of what is to be built, and actually building it, rather than just talking about it or drawing diagrams about it, realizing of course that these other tasks are also important. In my view, these three specializations overlap with one another and are not mutually exclusive. I've explained to executives the last few years that the line that had been traditionally drawn by the technology industry between software development and data work is no longer well defined, partially because the tooling between these two spaces has converged, and partially because, as a result of this convergence, data work itself has largely become a software development effort. However, since traditional data practitioners typically don't have software development backgrounds, and vice versa, I help meet this gap. 

Media: What is an interesting project that you are currently working on with SPR?

Gfesser: Just recently, I published the first post in a multi-part case study series about the earlier mentioned data platform that my team and I implemented in AWS from scratch this past year for the CIO of a Chicago-based global consultancy. This platform consists of data pipelines, data lake, canonical data models, visualizations, and machine learning models, to be used by corporate departments, practices, and end customers of the client. While the core platform was to be built by the corporate IT organization run by the CIO, the goal was that this platform would be used by other organizations outside corporate IT as well to centralize data assets and data analysis across the company using a common architecture, building on top of it to meet the use case needs of each organization. As with many established firms, use of Microsoft Excel was commonplace, with spreadsheets commonly distributed within and across organizations, as well as between the firm and external clients. Additionally, business units and consultancy practices had become siloed, each making use of disparate processes and tooling. So in addition to centralizing data assets and data analysis, another goal was to implement the concept of data ownership, and enable the sharing of data across organizations in a secure, consistent manner.

Media: Is there anything else that you would like to share about open source, SPR or another project that you are working on?  

 

Gfesser: Another project (read about it here and here) that I recently lead involved successfully implementing Databricks Unified Analytics Platform, and migrating the execution of machine learning models to it from Azure HDInsight, a Hadoop distribution, for the director of data engineering of a large insurer. All of these migrated models were intended to predict the level of consumer adoption that can be expected for various insurance products, with some having been migrated from SAS a few years prior at which time the company moved to making use of HDInsight. The biggest challenge was poor data quality, but other challenges included lack of comprehensive versioning, tribal knowledge and incomplete documentation, and immature Databricks documentation and support with respect to R usage at the time (the Azure implementation of Databricks had just been made generally available a few months prior to this project). To address these key challenges, as a follow-up to our implementation work I made recommendations around automation, configuration and versioning, separation of data concerns, documentation, and needed alignment across their data, platform, and modeling teams. Our work convinced an initially very skeptical Chief Data Scientist that Databricks is the way to go, with their stated goal following our departure to be migration of their remaining models to Databricks as quickly as possible.

 

---

See all of my responses to media queries here.

Media Query Source: Part 21

The responses I provided to an "anonymous" media outlet series of queries on December 4, 2019:

Media: Technical debt is the added cost that's incurred when software is developed quickly with little or no regard to quality and must be reworked to reach its intended objectives. Looking for experts who can answer some or all of the following questions:

1. Why is technical debt bad? 

2. What is an effective method for reducing technical debt without killing operations? 

3. What's the best way to plan a software project that ensures technical debt is either minimized or eliminated? 

4. Is there ever a time when technical debt is actually useful? 

 

Gfesser: Absolutely. As such, the definition of technical debt provided for this media query ("the added cost that's incurred when software is developed quickly with little or no regard to quality and must be reworked to reach its intended objectives") can be a bit of a misnomer, depending on the audience.

 

I've often used the incurring of technical debt as a strategy in order to develop code that fulfills application needs from a functional perspective, knowing that it will need to be revisited for refactoring for a variety of reasons. For example, code may not be fully configurable with respect to environment differences or user needs, or fully enabling DevOps processes. But the lack of these code attributes doesn't mean the code is functionally incorrect. 

 

When developers use this strategy, it does not mean that they have low regard to quality. In fact, code should ideally be written in a way which permits the addition of these missing attributes at a later point, meaning that even though technical debt is being incurred, it doesn't mean that code quality is bad. This just means that some attributes of the code will be worked in at a later point.

 

The danger of using this strategy, however, is that not all stakeholders will understand how it works, or the implications of making use of it. For example, business stakeholders with minimal technology backgrounds may not understand that just because an application is functionally correct that it may not necessarily be an ideal foundation on which to add additional code down the road.

 

One key implication in using this strategy is that technical debt needs to be periodically addressed, and cannot be forgotten about or consistently deprioritized by stakeholders over a lengthy period of time, since the more time which passes before it is addressed can often lead to greater challenges as the code base can increase in complexity in the meantime.

 

It's a good idea to schedule either the periodic tackling of technical debt, say every X number of Sprints, or schedule around a particular release schedule. I've often, for example, knowingly incurred technical debt prior to the release of an MVP (minimum viable product), only to schedule the tackling of technical debt in an incremental manner that coincides with either the MVP or a release shortly following up to it.

 

Another example is the postponement of addressing application performance. I've followed the philosophy of Donald Knuth that "Premature optimization is the root of all evil", because spending too much time on performance improvement before an application is actually used (or before application usage is known) deviates valuable time that could otherwise be spent on functional correctness.

 

With respect to some application aspects such as performance, it's important to understand that while the architecture of a given application or system can evolve, in order to minimize disruption it is imperative to choose an architecture which can evolve well.  So with respect to performance, for example, while the development team may not know how much data might be processed down the road, the architecture should be able to scale well in order to prevent ripple effects in the code, or worst case, a completely new architecture within a relatively short period of time.

5. Is there anything else you would like to add?

---

See all of my responses to media queries here.

New Book Review: "Designing Data-Intensive Applications"

New book review for Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, by Martin Kleppmann, O'Reilly, 2017, reposted here:

Kleppmann mentioned during his "Turning the Database Inside Out with Apache Samza" talk at Strange Loop 2014 (see my notes) that he was on sabbatical working on this book, and while waiting quite some time for it to be published, I ended up experimenting with his Bottled Water project as well as Apache Kafka (which was only at release 0.8.2.2 at that point in time). Other reviewers are correct that much of the material included in this book is available elsewhere, but this book is packaged well (although still at 550-pages and heavyweight), with most of the key topics associated with data-intensive applications under one roof with good explanations and numerous footnotes which point to resources providing additional detail.

Content is broken down into 3 sections and 12 chapters: (1) foundations of data systems, which covers reliable, scalable, and maintainable applications, data models and query languages, storage and retrieval, and encoding and evolution, (2) distributed data, which covers replication, partitioning, transactions, the trouble with distributed systems, and consistency and consensus, and (3) derived data, which covers batch processing, stream processing, and the future of data systems. The latter 6 chapters are weighted more heavily, with chapter 9 on consistency and consensus, and chapter 12 on the future of data systems, the most lengthy with each comprising about 12% of the book.

Some potential readers might be disappointed that this book is all theory, but while the author does not provide any code he discusses practical implementation and specific details when applicable for comparisons within a product category. In my opinion, the last chapter is probably the most abstract simply because it explores ideas about how the tools covered in the prior two chapters might be used in the future to build reliable, scalable, and maintainable applications. Similiary, the chapter on the opposite end of this book sets the stage well for any developer of nontrivial applications with its section on thinking about database systems and the concerns around reliability, scalability, and maintainability.

About a year ago, I recall an executive colleague responding to me with a quizzical look when I mentioned that tooling for data and application development is converging over time, and just a few months prior I mentioned in a presentation to developers that transactional and analytical capabilities are being provided more and more by single database products, with one executive in the audience shaking his head in disagreement that kappa rather than lambda architectures are the way to go. Kleppman mentions that we typically think of databases, message brokers, caches, etc as residing in very different categories of tooling because each of these has very different access patterns, meaning different performance characteristics and therefore different implementations.

So why should all of this tooling not be lumped together under an umbrella term such as 'data systems'? Many products for data storage and processing have emerged in recent years, optimized for a variety of use cases and no longer neatly fitting into traditional categories: the boundaries between categories are simply becoming blurred, and since a single tool can no longer satisfy the data processing and storage needs for many applications, work is broken down into tasks that can be performed efficiently on a single system that is often comprised of different tooling stitched together by application code under the covers.

In addition to the author's abundant and effective simple line diagrams that are reminiscent (although more sophisticated) of his earlier diagrams, one aspect that I especially appreciate is the nomenclature comparisons between products when walking through terminology. For example, at the beginning of chapter 6, the author specifically calls out the terminological confusion that exists with respect to partitioning. "What we call a 'partition' here is called a 'shard' in MongoDB, Elasticsearch, and SolrCloud; it's known as a 'region' in HBase, a 'tablet' in Bigtable, a 'vnode' in Cassandra and Riak, and a 'vBucket' in Couchbase. However, partitioning is the most established term, so we'll stick to that."

In addition, Kleppmann walks through differences between products when the same terminology is being used, which can also lead to confusion. For example, in chapter 7 the author provides a great 5-page discussion on the meaning of "ACID" (atomicity, consistency, isolation, and durability), which was an effective reminder to me that while this term was coined in 1983 in an effort to establish precise terminology for fault-tolerance mechanisms in databases, in practice one database's implementation of ACID does not equal another's implementation. "Today, when a system claims to be 'ACID compliant', it's unclear what guarantees you can actually expect. ACID has unfortunately become mostly a marketing term."

If you've ever found yourself confused about the concept of "consistency", the author offers a sanity check that your confusion is warranted, not only because the term is "terribly overloaded" with at least four different meanings, but because "the letter C doesn't really belong in ACID" since it was "tossed in to make the acronym work" in the original paper, and that "it wasn't considered important at the time." The reality is that "atomicity, isolation, and durability are properties of the database, whereas consistency (in the ACID sense) is a property of the application. The application may rely on the database's atomicity and isolation properties in order to achieve consistency, but it's not up to the database alone."

An later in chapter 9 where he discusses consistency and consensus, the author provides a great sidebar on "the unhelpful CAP theorem". As Kleppmann later comments, "the CAP theorem as formally defined is of very narrow scope: it only considers one consistency model (namely linearizability) and one kind of fault (network partitions, or nodes that are alive but disconnected from each other). It doesn't say anything about network delays, dead nodes, or other trade-offs. Thus, although CAP has been historically influential, it has little practical value for designing systems."

The author concludes in a sidebar by commenting that "all in all, there is a lot of misunderstanding and confusion around CAP, and it does not help us understand systems better, so CAP is best avoided." This is because "CAP is sometimes presented as 'Consistency, Availability, Partition tolerance: pick 2 out of 3'. Unfortunately, putting it this way is misleading because network partitions are a kind of fault, so they aren't something about which you have a choice: they will happen whether you like it or not...A better way of phrasing CAP would be 'either Consistent or Available when Partitioned'. A more reliable network needs to make this choice less often, but at some point the choice is inevitable."

While the second section of this text on distributed data was most beneficial to me, the third section on derived data was least beneficial, mainly because I'm already familiar with these topics from recent readings and experience, and because I needed to refamiliarize myself with the content discussed in the second section. However, the author presents derived data well, and I certainly do not recommend skipping this section. As Kleppmann comments, the issues around integrating multiple different data systems into one coherent application architecture is often overlooked by vendors who claim that their product can satisfy all of your needs. In reality, integrating disparate systems (which can be grouped into the two broad categories of "systems of record" and "derived data systems") is one of the most important things that needs to be done in a nontrivial application. I highly recommend this text.

Media Query Source: Part 11

The responses I provided to an "anonymous" media outlet series of queries on February 8, 2018: 

Media: Why are microservices becoming so popular with large brands like Netflix? Are they the future of app building? Are smaller companies seeing the same benefits? Do microservices bring the same benefits to small scale projects with small teams? What are the downsides to using microservices with smaller teams?

Gfesser: Microservices can offer significant flexibility to development teams, because application logic for each is intended to be limited to a discrete business domain or functional responsibility. The idea is that changes to each will have relatively little affect on other microservices, leading to the ability to test and deploy with less need to coordinate across development teams. Building applications in discrete units can have a significant upside relative to building monolith applications that combine functionality and data from multiple domains, because changes to monoliths are not isolated and tend to affect the entire application by requiring more testing and coordination between developers.

As with many application development practices, the decision to make use of microservices will partially depend on factors such as team size, company culture and organization, how and to what extent applications are used, and team familiarity of the problem space. And determination as to whether microservices should be developed is not an all-or-nothing proposition: monoliths and microservices can oftentimes work together to provide what is needed. Generally speaking, microservices tend to provide benefits for larger development teams presumably working on larger applications, but can be challenging to build for matrixed organizations, since microservices teams benefit from being cross-functional and having little dependence on other teams.

Development using microservices is not a silver bullet. While well designed microservices have much to offer the enterprise, it can be difficult to initially determine boundaries between different domains. Additionally, introducing independently running microservices brings complexity to the enterprise, especially when communication needs to take place between two or more microservices. Generally speaking, while microservices can help enable decreased time to market in the long-term, in the short-term it can make sense to make use of well-designed monolith applications that can be broken up into microservices at a later point, especially when domains become better understood. Oftentimes, it is difficult to predict how new functionality will be used and to what extent in terms of traffic volume. Prematurely making use of microservices, for example, can increase unnecessary developer workload at an early stage.

Keep in mind that Netflix didn't use microservices at the outset - it was a journey. Other organizations can benefit from the lessons that were learned by Netflix and others, but all development teams should expect to have their own journeys.

---

See all of my responses to media queries here.

New Book Review: "Spring Boot Messaging"

New book review for Spring Boot Messaging: Messaging APIs for Enterprise and Integration Solutions, by Felipe Gutierrez, Apress, 2017, reposted here:

When I first ran across this text, I debated whether it would make sense to acquire a copy for a recent client project of mine. As a relatively early adopter of Spring Boot in early-2015 before any books were available, and as someone who has been using Spring frameworks for quite some time, I questioned the possible value of working through a book. Don't get me wrong. When it makes sense, I often use books as starting points when available, and was known for quite some time amongst my colleagues for having read through (over a single weekend) the 700-page first edition of "Spring in Action" after being told by the technical staff at my then new employer that use of Spring was mandated for Java development.

Anyone new to the Spring Boot ecosystem needs to understand that "Spring Boot Messaging" is not the name of a Spring project. This book is an attempt by Gutierrez to bring a discussion of all messaging related Spring frameworks under one roof so to speak. And this aspect appealed to me. However, when I cracked open this book and flipped through it for the first time, my negative memories of circa Dot-com bubble technology texts surfaced. I'm not sure specifically why: perhaps the choice in fonts, perhaps the attempt at covering too many frameworks at once, perhaps the inclusion of code that seemed at first glance to be too disconnected from the discussion, or perhaps my realization that Spring Cloud Stream coverage is limited to only one chapter.

Soon afterward, I readied my copy of this book for return shipment, and dove into Spring reference materials and what the community provided with regard to examples and advice. In the meantime, I refrained from moving forward with the return in the case that I ended up determining that my initial judgment was faulty (it was). After understanding how the covered Spring projects are all interrelated, and exploring the code that the author provides in GitHub, my perspective changed. The coverage is actually decent, and for what it's worth this is really the only book available on these projects. And Spring Integration and related projects on which Spring Cloud Stream are built are in my opinion some of the most challenging Spring frameworks (the others being Spring Security and Spring Batch).

My focus here was on chapter 5 ("AMQP with Spring Boot"), chapter 8 ("Messaging with Spring Integration"), and chapter 9 ("Messaging with Spring Cloud Stream"), because I was looking to potentially adopt Spring Cloud Stream for a recent client project of mine, but wanted to understand some of the Spring magic that is often missing from community discussions. The higher level abstraction that Spring Cloud Stream provides on top of Spring AMQP, Spring Rabbit, and Spring Integration is convenient, but as with other frameworks it quickly becomes important to understand the underlying mechanisms. Expect to spend some significant time with the provided code examples, followed by what the Spring projects themselves and the community provide.

Media Query Source: Part 7

The responses I provided to an "anonymous" media outlet series of queries on August 8, 2017: 

Media: How to Explain Microservices in Plain English.

My responses addressed the more detailed questions as follows: 

Media: How do you define microservices succinctly?

Gfesser: A microservice can be described as an independently running application that is responsible for a specific domain (often referred to as a "bounded context"). For example, a health insurance firm is likely to consider domains such as claim, provider, and client, and to create a microservice for each. Another application that needs claim presentation, logic, or data should make use of the claim microservice because it is responsible for this particular domain, rather than make use of one or more other applications elsewhere in the enterprise.

Media: How do you explain its benefits?

Gfesser: When presentation, logic, and data about a specific domain is made available from a single location, there is no question as to where to obtain this information, as there can otherwise tend to be redundancy in the enterprise. In addition, lack of dependencies between microservices can help enable DevOps teams to develop, test, deploy, and maintain each separately from one another, leading to decreased time to market.

Media: What are the common misunderstandings about microservices?

Gfesser: There are three common misunderstandings that immediately come to mind. The first is that use of microservices is the same as a circa-2005 practice called SOA (service oriented architecture). One of the key distinguishing factors between these two is that use of SOA tried to reuse as much code in the enterprise as possible. However, extensive reuse tends to mean that significant dependencies also exist. The second is that use of microservices is a silver bullet. While well designed microservices have much to offer the enterprise, it can be difficult to initially determine boundaries between different domains. Additionally, introducing independently running microservices brings complexity to the enterprise, especially when communication needs to take place between two or more microservices.

And this leads to a related third misunderstanding, namely that development of new systems should make use of microservices from the start. Generally speaking, while microservices can help enable decreased time to market in the long-term, in the short-term it can make sense to make use of well-designed traditional applications (often referred to as "monoliths") which can be broken up into microservices at a later point, especially when domains become better understood. Oftentimes, it is difficult to predict how new functionality will be used and to what extent in terms of traffic volume. Prematurely making use of microservices, for example, can increase unnecessary developer workload at an early stage.

---

See all of my responses to media queries here.

New Book Review: "Microservices vs. Monoliths"

New book review for Microservices vs. Monoliths: The Reality Beyond the Hype, by Chris Richardson, Dan Haywood, Mike Gehard, Matt McLarty, Mark Little, and Adrian Cockcroft, C4Media (Publisher of InfoQ.com), 2017, reposted here:

This book brings together several previously published articles, blog posts, and presentations by individuals who have implemented successful, maintainable systems across the spectrum of monoliths to microservices. The first entry by Chris Richardson lays out guidelines for building a microservice architecture around the concepts of aggregates, event sourcing and CQRS (command query responsibility segregation). The second entry by Dan Haywood discusses "modular monoliths" and why they are a better option than microservices when working within a complex domain that does not need to support internet-scale traffic. The third entry by Mike Gehard walks through a recent migration from a monolith to microservices that is intended to help enable readers to make smart decisions around this type of endeavor. The fourth entry by Mark Little presents a Q & A dialogue with a panel of 5 developers and architects that discusses microservices in practice. The fifth entry by Adrian Cockcroft focuses on some underlying technology advancements that are creating a shift to event driven functions and improvements in time to value. Sandwiched in-between the third and fourth entries of this book is a brief 3-page article (in the form of a CA advertisement) that addresses the need for a common language for microservice architecture.

If you are looking for a place to start understanding some of the decision making will be required in your microservice architecture journey, I recommend starting here, but reading in a different order if you have never previously tackled this subject matter. Start with the second, third, and fifth entries entitled "In Defense of the Monolith", "The Journey from Monolith to Microservices: A Guided Adventure", and "Evolution of Business Logic from Monoliths through Microservices, to Functions", followed by the first entry entitled "Developing Transactional Microservices Using Aggregates, Event Sourcing, and CQRS", the fourth entry entitled "Virtual Panel: Microservices in Practice", and the 3-page article entitled "A Common Language for Microservice Architecture". The idea here is to read about the spectrum of monoliths to microservices in order so-to-speak, and then look to start looking at some specific ways to build microservices as presented by Chris Richardson (the founder of the original CloudFoundry.com) as well as getting a feel for what the community is saying about the place of monoliths and microservices and the direction things seem to be going. While most of the articles in this text are relatively technology agnostic, I like the fact that Mike Gehard (from Pivotal Labs) presents within the context of Spring Boot and Spring Cloud, and that the panelists explicitly mention technologies in the microservices space.

In my opinion, the first three entries of this book are especially well done (pointing to a possible reason why these are included first), as is the panel discussion that presents a lot of discussion points in a small amount of space. Some of the discussion coverage especially of interest to me are the challenges that developers face when writing microservices, the importance of using DDD (domain-driven design), how event-driven architecture can solve data consistency issues, how to choose between monoliths and microservices and the timing of this choice, how to best structure code projects to help enable a migration to microservices down the road, and the very opinionated do's and don'ts listed by the panelists. Some of my favorite don'ts: "Don't fall for vendor snake-oil - that's why SOA originally died a death" (James Lewis), "Don't throw away the years of developer experience and business logic already written. It's an evolution, not a new paradigm" (Martijn Verburg), "Don't approach microservices as a way to cut costs; Microservices is about enabling innovation and business outcomes through technology, not as a way to minimize operating costs, like traditional IT has been treated for decades" (Christian Posta), and "If you have trouble with CI/CD, APIs, DevOps, self-service platforms, or self-service teams, then don't force a microservices architecture; get the principles, practices, and organizational learning systems down first. Doing microservices isn't the goal; fast moving, innovative teams is the goal. Get the foundational pieces in place first" (Christian Posta).

Some of my favorite do's: "Prefer rapid remediation and canary deploys over integration testing" (James Lewis), "Build three prototype services that communicate with each other and figure out how to do all of the nonfunctional requirements like security, service discovery, health monitoring, back pressure, failover etc., before you go and build the rest" (Martijn Verburg), "Put developers and feature teams as close to their customer (or even in their customer's shoes) so they see the pain directly from the systems the teams build" (Christian Posta), and "Do pay attention to data. Data is the lifeblood of a company. When building services, pay attention to use case boundaries, transaction boundaries, consistency issues, and data processing (stream, storage, etc.)" (Christian Posta). Outside these do's and don'ts, I also appreciated the following comments: "Splitting a bloated monolith into smaller overcomplicated monoliths will only make the situation worse. In brownfield projects it is crucial to remove the cargo cult based patterns first, and re-learn the domain concepts. Splitting a lean monolith into independent units becomes a fully optional task." (Adam Bien), and "Forget about all modularization attempts from the past. Keep your code simple. Thin WARs are the ultimate module. Be paranoid and assume that the whole infrastructure around your service can and will fail. Test the behaviour in failure case. Provide the simplest possible solution (should be classes, not frameworks)" (Christian Posta). One of the only areas of the panel discussion with which I disagree are the references to the Gartner Hype Cycle, as technologies should not be expected to follow the curve over time, since many technologies come and go after making a relatively brief appearance.

New Book Review: "Spring Microservices"

New book review for Spring Microservices, by Rajesh RV, Packt Publishing, 2016, reposted here:

The mix of theory and practice that this book provides is among the best I have ever run across, although those looking to potentially work through this book should be aware that a number of issues start surfacing in Chapter 5. Hopefully these issues are resolved in the second edition due out in October 2017, but in the meantime this book is second to none in the Spring Boot microservices space, not to mention amongst the best in the general microservices space.

While all 10 chapters provide theory, Chapter 2 and Chapters 5 through 9 also provide development exercises. The discussions and exercises build upon one another as the text progresses, starting with a 100-page introduction to microservices in Chapters 1 and 3 and a Spring Boot introductory exercise in Chapter 2, and a monolith-to-microservices case study in Chapter 4 that provides background for the exercises covered in the next 200-pages. The last chapter (Chapter 10) provides some needed discussion on the microservices development lifecycle, but is in my opinion a bit rushed, so readers should expect the need to explore this topic further using other resources.

While I have been a heavy user of open source software and libraries for many years, the short lifespans of some of the Spring technologies used throughout this book are especially apparent for someone like me who prefers to use the latest releases whenever possible. For example, I initially had no issues updating the projects provided by the author from Spring Boot 1.3.5.RELEASE (May 2016) to 1.5.2.RELEASE (March 2017). But by the time I got to Chapter 8 my decision was to start re-architecting a real product that I had recently built with my development team, and doing so became more difficult as the code base was much more complicated than what is presented by the author.

The one exception was my upgrading of Spring Cloud Brixton.RELEASE (May 2016) to Camden.Release (September 2016) after running into a number of issues with technologies such as Zuul (which is covered alongside other technologies such as Eureka in Chapter 5). However, as my real product extensively used Spring Security as is the case with all serious Spring Boot applications, the complete absence of Spring Security in this book was a surprise. Readers new to Spring in general should expect the need to extensively explore Spring Security apart from this material as it can be one of the most complex Spring projects, and in my experience the community tends to offer very poor guidance.

Other than differences in releases, readers should be aware of other issues in the exercises. While not an issue per se, my qualms began with Chapter 2 where the author begins running Spring Boot applications using "java -jar" on the command line. It wasn't until I started re-architecting and Dockerizing my real product as part of working through Chapter 8 that I realized the author was probably avoiding use of "mvn clean spring-boot:run" because of the way Docker works.

Some issues are relatively more trivial, such as SMTP configuration that mistakenly uses port 2525 rather than the default port of 25 in Chapter 2, although a book written for those new to Spring in general should mention in Chapter 5, for example, that use of Maven profiles are another common option to externalize configuration. Interestingly, some of the content that the author provides is also out of order, the most glaring example of which is toward the tail end of Chapter 5 when setting up Eureka is done after a 16-step process outlining how to access the example configuration server from clients, which itself depends on Eureka being up and running.

Please take my advice and perform the Chapter 8 exercise in Linux after working through the 6-part "Get Started with Docker" series of exercises on the Docker website in Linux as well, as Docker for Windows is a mess right now and the author just does not explain Dockerization very well. And as a long-time user of Spring, I was not aware that not using the spring-boot-starter-parent Maven dependency would lead to incomplete manifest files and therefore the inability to start applications correctly.

So in the end, why did I chose to grant this book 5 stars? Because the author provides a lot of the microservices theory that is skipped elsewhere, and includes practical application as part of the discussion. That said, I suggest additionally checking out 4 freely available microservices books from O'Reilly that I also recently read: "Microservices AntiPatterns and Pitfalls", "Reactive Microservices Architecture", "Microservices vs. Service-Oriented Architecture", and "Migrating to Microservices Databases".

New Book Review: "Migrating to Microservice Databases"

New book review for Migrating to Microservice Databases: From Relational Monolith to Distributed Data, by Edson Yanaga, O'Reilly, 2017, reposted here:

This book addresses a much-needed topic in microservices architecture: the database in the context of microservices. However, after reading several other books on microservices architecture, I found that much of the early content here does not really cover what will likely be expected by readers, based on the title. In my view, the one exception is the fact that the author addresses the monolith-first approach (also called the strangler pattern), and quotes Martin Fowler as writing that "almost all the successful microservice stories have started with a monolith that grew too big and was broken up", and "almost all the cases I've heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble".

Yanaga furthers these thoughts: "Having a stable monolith is a good starting point because one of the hardest things in software is the identification of boundaries between the domain model - things that change together, and things that change apart. Create wrong boundaries and you'll be doomed with the consequences of cascading changes and bugs. And boundary identification is usually something that we mature over time. We refactor and restructure our system to accommodate the acquired boundary knowledge. And it's much easier to do that when you have a single codebase to deal with, for which our modern IDEs will be able to refactor and move things automatically. Later you'll be able to use these established boundaries for your microservices."

"The usual candidates for the first microservices in your new architecture are new features of your system or changing features that are peripheral to the application's core. In time, your microservices architecture will grow just like a strangler fig tree, but we believe that the reality of most companies will still be one, two, or maybe even up to half-dozen microservices coexisting around a monolith." And following this train of thought, the author explains that the challenge of choosing which piece of software is a good candidate for a microservice requires Domain-Driven Design (DDD) knowledge, a discussion which also helps set the stage for the material presented later in the book.

Readers like me who have been using database change management tooling such as Liquibase for quite some time can safely skip Chapter 3, but one thing worth mentioning here is that the author provides a good sidebar at the end of Chapter 1 around the topic of nonrelational database products. As Yanaga comments, while nonrelational database products have often been successfully implemented and used, he still recommends that readers begin their microservices journeys on the safe side by first using the existing relational databases around which their monoliths have been built, and then determining whether use of another type of database technology might be better after integrating a first microservice. Simply stated, the microservices journey is not trivial and probably will not be the best time to introduce a new database technology.

In my view, Chapter 5 (in combination with Chapter 4, which discusses CRUD and CQRS) was the most beneficial for my microservices journey. In this chapter, the author presents 9 different database migration strategies: (1) shared tables, (2) database views, (3) database materialized views, (4) database triggers, (5) transactional code, (6) ETL tooling, (7) data virtualization, (8) event sourcing, and (9) change data capture (CDC). Across this presentation, the author provides quite a few helpful considerations that need to be made as part of the decision making process. Potential readers might be interested in knowing that the author introduces a new open source project called Debezium as part his discussion on CDC. As a PostgreSQL user who experimented last year (2016) with an open source project called Bottled Water, I was particularly interested in this solution after discovering that PostgreSQL support was added just a few weeks ago (February 2017) with version 0.4.0.

New Book Review: "Microservices vs. Service-Oriented Architecture"

New book review for Microservices vs. Service-Oriented Architecture, by Mark Richards, O'Reilly, 2016, reposted here:

Unlike some other books focused on microservices architecture, Richards specifically investigates microservices architecture in the context of service-oriented architecture (SOA) that came to dominate the technology industry circa 2005. As the author explains in his preface, "dozens of best practices for implementing SOA emerged at that time, as well as a plethora of third-party products to help companies implement SOA", but "unfortunately, companies learned the hard way that SOA was a big, expensive, complicated architecture style that took too long to design and implement, resulting in failed projects that drove SOA out of favor in the industry". It would be easy to view the current acceptance of microservices architecture in much the same light if distinctions were not made between SOA and microservices architecture, so Richards provides a much needed text that compares the two.

In my earlier review of Sam Newman's "Building Microservices: Designing Fine-Grained Systems", I mentioned that the "Microservices and SOA" sidebar of Martin Fowler's "Microservices" article indicates that the "common manifestation of SOA has led some microservice advocates to reject the SOA label entirely" and "others [to] consider microservices to be one form of SOA, perhaps service orientation done right", but "the fact that SOA means such different things means it's valuable to have a term that more crisply defines this architectural style". Newman clarifies the microservice approach by explaining it "as a specific approach for SOA in the same way that XP or Scrum are specific approaches for Agile software development", which "has emerged from real-world use, taking our better understanding of systems and architecture to do SOA well", but I like the simple, focused approach that Richards presents here.

After first taking a look at the world of service-based architectures from the standpoint of service contracts (including versioning), service availability, security, and transactions, the author expectantly asks whether there is just too much complexity involved. While Richards agrees that service-based architectures are complex, he also points our that adopting such an approach involves trade-offs. While new issues are introduced, service-based architectures also provide additional capabilities that will help enable greater development team productivity, more reliable and robust applications, overall cost reduction, and decreased time to market.

However, SOA and microservices architecture vary greatly in terms of service characteristics, so the author follows up these statements by walking the reader through the differences of these approaches over the next couple chapters, addressing topics such as service taxonomy, service ownership and coordination, service granularity, granularity and pattern selection, component sharing, service orchestration and choreography, middleware versus API layer, accessing remote services, application scope, heterogeneous interoperability, and contract decoupling. While I have read several different books focused on microservices architecture, I will be recommending this book above all others of this genre at this point, due to its focus and terseness.

New Book Review: "Reactive Microservices Architecture"

New book review for Reactive Microservices Architecture: Design Principles for Distributed Systems, by Jonas Bonér, O'Reilly, 2016, reposted here:

This text provides an introductory look at Reactive microservices architecture, written by Jonas Bonér, co-author of the Reactive Manifesto. After a brief look at the general concept of microservices, Bonér quickly shifts focus to Reactive principles and why these are in no way new, having been proven and hardened since they were conceived more than 40 years ago by individuals ahead of their time. The author explains that the most important parts that define a Reactive microservice are (1) to isolate all the things, (2) to act autonomously, (3) to do one thing, and do it well, (4) to own your state, exclusively, (5) to embrace asynchronous message passing, and (6) to stay mobile, but addressable.

While discussing these microservice traits, Bonér delves into a number of other related concepts such as divide and conquer, share-nothing architecture, bulkheading, continuous delivery, domain-driven design, data streaming, and location transparency. For the remainder of the text, the author discusses microservices working together as systems, exploring a number of concepts in the process such as discovery, coordination, security, replication, data consistency, failover, deployment, and integration with other systems. A good portion of this relatively terse 50-page text is consumed by drawings somewhat reminiscent of those found in works by Martin Kleppmann, greatly enhancing the material.

My reading of this text follows my many years in the development space and a recent reading of "Building Microservices: Designing Fine-Grained Systems" by Sam Newman. If you are new to microservices, I recommend reading the report discussed here prior to investing the time needed to wade through the presentation by Newman, as it is roughly 5 times in length. And at some point, you will need to get some hands-on experience as an architect after becoming acquainted with the theory, as execution is always harder than strategy. While the author does mention a handful of tools in passing, and is the inventor of the Akka project, this report is technology agnostic with regard to the discussed concepts and you will need to access other materials to explore implementation details.

New Book Review: "Microservices AntiPatterns and Pitfalls"

New book review for Microservices AntiPatterns and Pitfalls, by Mark Richards, O'Reilly, 2016, reposted here:

As explained in the preface, the intention of the author is to introduce several of the more common antipatterns and pitfalls that emerge when using microservices. The goal here is to help readers avoid costly mistakes by helping in the detection of when antipatterns or pitfalls take place, as well as the understanding of techniques and practices for preventing these occurrences. As pointed out by Richards in the outset, the intention here is to discuss the more common antipatterns and pitfalls rather than attempt to cover all of them in a report of roughly 55-pages.

The author covers four antipatterns and six pitfalls: the data-driven migration antipattern, the timeout antipattern, the "I was taught to share" antipattern, the reach-in reporting antipattern, the grains of sand pitfall, the developer without a cause pitfall, the jump on the bandwagon pitfall, the static contract pitfall, the are we there yet pitfall, and the give it a rest pitfall. The majority of these presentations are able to make a point in roughly 5 to 6 pages each, and much of the content is consumed by diagrams which adds considerably to the discussion.

In my view, the chapters on all four microservices antipatterns will provide value to most architects and developers, as will the first two chapters on microservices pitfalls, but the last three chapters can likely be safely skipped due to the incomplete discussion of the static contract pitfall and the relatively short discussions of the are we there yet and give it a rest pitfalls. Unfortunately, the author just constrains the static contract pitfall to two types of versioning (header and schema) without even mentioning options such as URI versioning, and the last two pitfalls are given only cursory consideration.

The data-driven migration antipattern is likely one of the most important discussions in this text, especially for architects and developers lacking significant database experience, or for those who want to understand some of the data implications that should be considered as part of a migration to microservices. Not only are the diagrams especially well done, Richards also quotes a fellow O'Reilly author who once told him that "data is a corporate asset, not an application asset". Well said.

New Book Review: "Making Sense of Stream Processing"

New book review for Making Sense of Stream Processing: The Philosophy Behind Apache Kafka and Scalable Stream Data Platforms, by Martin Kleppmann, O'Reilly, 2016, reposted here:

My first exposure to author Martin Kleppmann was at his Strange Loop 2014 talk in St Louis, Missouri entitled "Turning the Database Inside Out with Apache Samza". The reason this talk is of relevance is because Samza is a distributed stream processing framework developed at LinkedIn that is in the words of Kleppmann "a surreptitious attempt to take the database architecture we know, and turn it inside out", using Apache Kafka at its core for use as a distributed, durable commit log.

After discussing events and stream processing in Chapter 1, and using logs to build a solid data architecture in Chapter 2, the author discusses integrating databases and Kafka with change data capture (CDC) in Chapter 3, and the "Unix philosophy" of distributed data in Chapter 4, followed by the text version of the Strange Loop 2014 talk in Chapter 5. This freely available text is probably one of the most organized that I have seen from O'Reilly as of late, although readers will need to keep in mind that the last chapter comprises material from 2 years ago. Other than organization and readability, another aspect of this book that other architects such as myself will probably appreciate is the high number of diagrams. While many of these diagrams are drawn at a high level, just remember that this presentation covers practical architecture, with no ivory tower nonsense.

With the probable expectation that some might view stream processing as ivory tower nonsense, however, the following comments are provided in the foreward to this book: "Whenever people are excited about an idea or technology, they come up with buzzwords to describe it. Perhaps you have come across some of the following terms, and wondered what they are about: 'stream processing', 'event sourcing', 'CQRS', 'reactive', and 'complex event processing'. Sometimes, such self-important buzzwords are just smoke and mirrors, invented by companies that want to sell you their solutions. But sometimes, they contain a kernel of wisdom that can really help us design better systems."

"In this report, Martin goes in search of the wisdom behind these buzzwords. He discusses how event streams can help make your applications more scalable, more reliable, and more maintainable. People are excited about these ideas because they point to a future of simpler code, better robustness, lower latency, and more flexibility for doing interesting things with data. After reading this report, you'll see the architecture of your own applications in a completely new light. This report focuses on the architecture and design decisions behind stream processing systems. We will take several different perspectives to get a rounded overview of systems that are based on event streams, and draw comparisons to the architecture of databases, Unix, and distributed systems."

If you are interested in and unfamiliar with this space, but don't have time to read the entire 170-page book, I recommend at least reading the first chapter. The author effectively walks through explanations of some of the buzzwords listed above, because the author realizes that some of the confusion related to technologies in this space "seems to arise because similar techniques originated in different communities, and people often seem to stick within their own community rather than looking at what their neighbors are doing."

After this discussion, the author lists distributed stream processing frameworks Samza, Storm, Spark Streaming, and Flink as alternatives to Kafka Streams, and comments about the interesting design differences (pros and cons) between these tools, but be aware that he points to the Samza documentation for additional detail, after also mentioning that all of these frameworks are mostly concerned with low-level matters and that work on high-level query languages for stream processing is currently in motion.

Web developers and architects will likely want to take notice of the second chapter, where Kleppmann walks through the long-term evolution of a case study web application that starts off simply enough in terms of architecture, but then grows with a proliferation of different tools used in combination with one another. "It's not that any particular decision we made along the way was bad. There is no one database or tool that can do everything that our application requires. We use the best tool for the job, and for an application with a variety of features that implies using a variety of tools. Also, as a system grows, you need a way of decomposing it into smaller components in order to keep it manageable. That's what microservices are all about."

"But, if your system becomes a tangled mess of interdependent components, that's not manageable either. Simply having many different storage systems is not a problem in and of itself: if they were all independent from one another, it wouldn't be a big deal. The real trouble here is that many of them end up containing the same data, or related data, but in different form."

Different techniques that make sure the data ends up in all the right places are then discussed, with all the inherent issues of each, followed by the simplest solution in the mind of the author: store all writes in a fixed order, and apply them in this fixed order to the various places they need to go. The rest of the chapter (25 pages) walks through examples of how logs are used in practice, concluding with a thought experiment: What if the only way to modify data in your service was to append an event to a log? And could all this activity take place through GET methods, negating the need for POST, PUT and DELETE?

While the fourth chapter dives into a bit more theory in the sense that the philosophy behind Kafka is important in assessing the tool, but not necessarily needed in practice, the presentation that the author provides here is worth reviewing because he walks through the reasons why some "old ideas" from Unix are very different from the design approaches of mainstream databases, and why these deserve more attention today.

It is probably the third chapter that might make some readers take notice, because it walks through a very practical example that involves an existing web application, and the desire to not rewrite the code in order to make use of Kafka. In short, instead of propagating data changes to all the different tools mentioned in the earlier web application case study, the example starts with the database as the starting point and propagates to a Kafka log, followed by propagation to these other tools. I'm currently extending a proof of concept I created for a recent client to implement this approach.

What is "Full-Stack"?

---

What is "full-stack" development? What does a "full-stack" developer do?

Typically, developers who feel that they can effectively program across all layers of an application refer to themselves as "full-stack". While I've heard this explanation on numerous occasions, I'm reminded of the current state of affairs in the application development space every time I go through another round of interviewing candidates for a development position, or a recruiter knocks on my door conjuring up the term "full-stack" once again, not understanding the implications of their use of the term. Note that "effectively program" means proficiency in a particular area. The need for (i.e. not simply the desire of) someone else to clean up code after the initial development is done, whether to make the UI "pretty" or otherwise, indicates probable lack of proficiency.

---

As I write this blog post, there are actually 3 things from my experience that immediately come to mind.

The first is a particular discussion with a prior architect colleague a few years ago who argued that essentially all that matters on a development project is the Java code. Of course, we were working on a Java project at the time, so substitute whatever core technology applies in your domain. In response, I listed a number of other technologies that were being used on the project, and raised my hand, forming a small circle with my fingers to visually describe the amount of Java code that existed relative to the rest of the project (and the enterprise, for that matter), which I subsequently described by forming a large circle with my arms outstretched around the top of my head. Their response: "Everything else other than the actual Java code is 'infrastructure'." And what did "infrastructure" mean to this individual? "Everything else exists to support the Java code."

(In response to readers, note that other technologies being used on a given project can include an increasingly growing and evolving area of the enterprise called "DevOps", associated with the automation of software delivery and infrastructure changes, tooling of which can fall into a variety of areas such as continuous integration, version control, configuration, and monitoring. The idea here was to limit scope of this discussion to the actual application stack, which admittedly comes from a traditional viewpoint. While it might be debatable whether this type of work should be included in a discussion of full-stack development, however, in my view what is not debatable is whether everything else in the enterprise simply exists to support application code, or in this case, a subset of the code that is written for an application that likely provides job security for some. It does not - and this was the point of my referencing this earlier discussion.)

For the sake of brevity, the second actually consists of multiple events I've chosen to group together. While I could write on this subject at length, what this area of thought boils down to is whether a developer can claim any degree of expertise - or any level of experience, for that matter - if they have simply been "exposed" to a particular technology on a project. Unfortunately, I have personally seen this claim on numerous occasions. Resumes stating in which technologies a "development environment" consisted, rather than in which technologies developers gained hands-on experience. Or laundry lists of technologies in which developers have been exposed to some degree, sometimes writing very limited code with the given technologies, or not incorporating any of them into their projects themselves and just resorting to what someone else "set up" for them.

The third is how one or more databases or "data stores" fit into the picture. I have seen red flags in this area for quite some time now amongst those who refer to themselves as full-stack developers, as well as those who interestingly refer to themselves as "back-end" or "service" developers. Verbal exchanges typically explore use of some type of data persistence layer. The issue however that can arise relatively quickly is what "used Hibernate" or "used JPA" actually means. Typically, what this means is that the developer used a data persistence layer that someone else set up. While past usage of such code is a positive for any project, what happens when an undecipherable exception is thrown by the framework, or the database underlying the persistence layer changes? And has the given developer actually ever designed any significant portion of a database?

---

Below is a simplified version of a high-level architecture diagram that I recently created for a client, showing solely the layers of the AngularJS / Java application portion of the architecture. The "front-end" architecture consists of a view layer, router layer, controller layer, and service layer, which together comprise what might be considered a traditional application user interface for presentation logic, although since this was an SPA (single page application), the view layer itself is really what comprises the HTML, CSS, and AngularJS directives that represent different UI components and other JavaScript libraries such as D3.js depicted in the full diagram. Note that my discussion of "the first cluster" below intentionally truncates the service layer depicted here.

The "back-end" architecture consists of a controller layer, service layer, persistence layer, and database or other data store. A REST API (i.e. RESTful services) is the entry point for this part of the architecture, although it should be noted that not all architectures follow this style. Since application security and other important features are managed at this level, equating these layers to "back office" or "back room" operations is a bit of a misnomer. However, it is accurate to state that the abstractions created for user experience are not managed by these layers of the architecture. Controller and service layers exist here for similar reasons as in the front-end: business logic is serviced by data similarly to how user actions are serviced by the UI.

So what does this diagram represent? It represents the complete, or full, development stack for the application portion of the development project. To be considered a true full-stack developer implies that a given developer is adept, or at least relatively proficient, in all of these areas. My argument is that very few, if any, developers are consistently proficient across all of these areas. It's not like I'm the only one voicing this opinion, and others have written about the myth of the full-stack developer, so this is to be expected. Of course, there are many who disagree, as readers can see from the aforementioned blog post, but commenters also reminded me of the following saying, with quotations often forgetting the second phrase:

Jack of all trades, master of none,
though oftentimes better than master of one.

From an application development perspective, what this adage implies is that it is better to have expertise in multiple areas, rather than just one area. And between the lines, simultaneously specializing in some areas over others should be viewed as a strength, as is recognizing when someone else is more aptly suited to work in a particular area. From my experience, there seem to be clusters of expertise for developers working on a tech stack, described in the following paragraphs.

The first cluster is in an area of the tech stack traditionally referred to as "the UI". Because much of these layers consist of JavaScript in an SPA, developers with expertise in this cluster are typically proficient from the view layer down to the "front-end" controller layer (to be differentiated from the "back-end" controller layer). And it should be noted that a significant share of developer "boot camps" currently in the marketplace seem to concentrate in this cluster. Now, with all of the clusters presented in this blog post, there will be some disagreement as to the boundaries for each. As it stands, I have chosen to create clusters with an overlap of one layer between each, because it seems from my experience that this is typically the case. Some developers might exhibit greater overlap, some developers none. The ideal is that the cluster of experience for a given developer is not smaller than one of these three.

 

 

 

 

 

 

 

 

 

 

 

Taking a look at the tech stack from the reverse end, the second cluster is an area traditionally referred to as "services". For the sake of simplicity, only a database is depicted in the following diagram, mainly because data is the object of a data persistence layer, but interfaces to external systems originating in the "back-end" service layer (to be differentiated from the "front-end" service layer) should also be expected. Note that the database layer is depicted in a different color because it is an area a bit different than the other layers in the sense that it is not typically (or should not be typically) developed from the perspective of programmatic code, and I have chosen to discuss this layer separately later in this blog post.

So what is left? The layers essentially comprising the glue between the "front-end" and "back-end", overlapping on the controller layer for the former, and the service layer for the latter. As I was thinking over what I wanted to present in this blog post, I realized that while expertise in this third cluster is often combined for a given developer with either expertise in the second cluster (such as myself) or expertise in the first cluster, I find it increasingly common to come across developers with expertise in this third cluster who neither have proficiency in the first cluster or second cluster. And this essentially means that such developers exhibit neither UI expertise or database expertise. In a traditional waterfall development approach, developers falling in this third cluster could probably survive, but with increasingly agile approaches user stories are more effectively tackled when there is also expertise in either the first or second clusters. The reason this is the case is because user stories typically provide functionality for either human or system users, and/or provide functionality that accesses data for reading, writing, or updating.

---

In my discussion of the second cluster of expertise, I mentioned that I would separately discuss the database or data store. It also should be noted that the persistence layer - whether it be an object-relational mapping framework such as Hibernate or JPA, or a JavaScript framework tied to a non-relational database - is typically tightly coupled to the data. Services can abstract much of this for developers concentrating on clusters "higher" in the tech stack, but at some point someone needs to work on this area of development. The exception might be RAD (rapid application development) frameworks out there such as Ruby on Rails or Groovy on Grails, but such frameworks are rare (or should be rare) in most enterprise architectures, since these are typically good solely for UI proof of concepts that are replaced down the road for actual long-term implementations.

In addition to understanding relational or non-relational databases at a conceptual and implementation level, developers working on the database need to understand the domain of the application as well as long-term implications of design decisions, since changes to data structures have ramifications for the data persistence layer. With some non-relational database products, implications for data structures might be lessened, but then expertise is pushed up the stack when it comes time to interpret the data. While I was initially planning to write a separate blog post to flesh out one of the questions I wrote for an earlier blog post of mine, this seems to be a good time to discuss.

In this earlier blog post, under software development question #13 of 20, I made the following statement:

Many software developers do not value data because they take it as a given as being available, think of it as "the easy part", do not understand it, or do not wish to work on it.

Let's briefly discuss these 4 points.

The data is already available. For relatively less involved application development projects, data is often already available. An example of such a project is a pure portal project in which an application is built to access this already available data (already available by other means, perhaps by only another set of users). Because the data is already available (at least most of the data is already available, as a small amount of data may need to be added for areas such as security), expertise in implementing the data store is not needed, just interpretation of some subset of the data for user consumption.

The data is "the easy part". Unlike the first point above where data is already available, this second point is subjective. The terms "easy" and "hard" are always subjective, regardless of source, because these are relative terms. In my experience, it is often the case that if data is not already available and a developer determines that it is the easy part, there is usually some other underlying reason for their confidence that falls under one of the two points below. Data is the lifeblood of most organizations, and decisions made with regard to data typically have longer term affects than decisions made in the programmatic code of higher layers because application code is typically easier to rewrite relative to restructuring data.

The data is not understood. This third point is very common. Developers in the first cluster of expertise explained earlier in this blog post typically fall into this category, depending on lower layers to interpret data for them. This reason for not valuing data is distinct from the other 3 because no judgement is being placed on whether the data is available, whether working with the data is difficult, or whether desire to work with the data exists. The data is simply misunderstood, and so the developer seeks to distance themselves from it. On one project earlier in my career, I recall mentioning to a developer that it is common that some developers do not understand data and database concepts. Their response was simply that data specialists do not understand object oriented concepts in the programmatic code. While not a good defense, this way of looking at things speaks to some of the data resistance that lives in some organizations.

A desire to work on the data does not exist. The verbiage of this fourth point is a diplomatic way of saying that a particular individual just doesn't want anything to do with the data. On a number of occasions, for example, I've unfortunately heard developers quip "I don't care about that" in regard to a number of areas, but the most common area is the data. In some sense, we've come full circle in this essay. In my introductory comments, I described use of my fingers, hands, and arms to talk about the place of a given technology within a given project in the enterprise. When it comes to data, sometimes zero digits is what expresses developer desire to work with data.

---

In thinking about your past, current, and future software development projects, it needs to be realized that as much as architects seek to decrease complexity, the very nature of the frameworks used in tech stacks are often in themselves complex, and so the ability to abstract this complexity can be limited to some degree. Because of the expertise needed to work on a given tech stack, coming to the realization that developers often frequently work in clusters of expertise (for specific reasons such as skill set, desire, or perceived company value) can only increase the health of a project, the technology industry, and your work advancing the progress of these two domains.

New Book Review: "Spring REST"

New book review for Spring REST, by Balaji Varanasi and Sudha Belida, Apress, 2015, reposted here:

Phenomenally practical guide for designing and developing REST services using the Spring Framework, Spring Boot, Spring Web MVC, and Spring Data. Unlike many technical books in the marketplace, this guide truly walks the reader step-by-step through the development process from square one, in this case to build a small application that demonstrates all of the core features modern REST services should provide.

In my case, working through this text followed a reading of "RESTful Java Patterns and Best Practices", by Bhakti Mehta, which provides a bigger picture view than is typically offered by other resources concentrating on specific technologies, and is among the very few available newer texts on the subject in the 2015 marketplace. Anyone newer to REST services might want to first check out this other book as well, keeping in mind that it was written from the perspective of the JAX-RS 2.0 API rather than Spring.

After an initial introduction to REST from a conceptual perspective, the authors offer a primer on Spring Web MVC for those who are not familiar with this particular Spring component, followed by an introduction to REST services by using Spring Boot to build a first application, and the design and development of a more complex application that will be worked on in an iterative fashion throughout the rest of the book, complete with error handling, documentation, versioning, paging, sorting, security, and testing.

In addition, the last chapter presents a small chapter on Spring HATEOAS and an even smaller 3-page appendix on installing cURL on Windows. The appendix might be considered convenient for some readers, but the material offered is essentially the same as on the cURL website. While I did not work through the last two chapters (which cover testing and Spring HATEOAS), I did work through 100% of the development covered in the rest of the book (the first eight chapters and the appendix), and for the most part the content offered is phenomenal.

For the most part, the first five chapters (the first 90 of approximately 175 pages) are error-free, with a couple dozen rather small coding issues that are typically mismatches between code and accompanying explanations. While I am surprised at myself for saying so, I actually did not end up minding these issues because it forced me to work them out. The publisher offers the code for this book in a ZIP file, but I do not recommend blindly using this code, because in my experience doing so inhibits the learning process. That said, I did take a look at the code on a couple occasions when issues were insurmountable.

Although the chapter on security (Chapter 7) has a couple issues, in my opinion, such as the magical replacement of @EnableWebSecurity with @EnableWebMvcSecurity in the OAuth 2.0 example, without explaining that this needs to be done, and without explaining the difference between these annotations (the Spring documentation explains that the latter adds AuthenticationPrincipalArgumentResolver), in general this chapter is very well put together, walking the reader through 6 popular approaches used for securing REST services, it is really only the chapter on documenting REST services (chapter 6) where the reader will likely need to turn to other sources.

In my case, I turned to the Springfox Reference Documentation for its guide on setting up Swagger UI. As explained by the guide, the Springfox suite of Java libraries are all about automating the generation of machine and human readable specifications for JSON APIs. The interesting thing, however, that often happens with published technical presentations presented by individuals such as Varanasi and Belida, however, is that information can get quickly outdated. In the case at hand, Swagger is now Springfox, the Maven dependencies now require Swagger2 rather than Swagger, and some of the packages have been completely renamed.

The out-of-the-box configuration of Swagger UI is actually quite good, once you understand how to incorporate it into the project, and quite easy (only two Maven dependencies and a single annotation is all that it takes to get you going). The confusion surrounding what is now being called Springfox can be surpassed relatively quickly, and the decision by the authors to include this chapter was a good one.

As an architect, I especially appreciated Chapter 1 ("Introduction to REST") for its explanations of resources, templates, HTTP methods and status codes, Chapter 3 ("RESTful Spring") for its presentation on the three different options to generate a new Spring Boot project, where I used the command line interface (CLI), Chapter 4 ("Beginning QuickPoll Application") for tying together all of the previously explained topics into workable code that is thoroughly annotated, and Chapter 8 ("Security") for transforming the example code base to use Basic Authentication, and later OAuth 2.0 (albeit with an in-memory token store for the authentication server and resource server).

New Book Review: "RESTful Java Patterns and Best Practices"

New book review for RESTful Java Patterns and Best Practices, by Bhakti Mehta, Packt Publishing, 2014, reposted here:

Concise 125-page, 6-chapter presentation on how to efficiently build scalable, reliable, and maintainable high performance REST services that is among the very few available newer texts on the subject in the 2015 marketplace. While this book reminds me somewhat of another Packt text that I read a couple years ago on a completely different subject entitled "Instant Drools Starter", because of its brevity and no-nonsense style, the difference here is that the author does cover considerable ground with its small footprint. Unlike some other publishers which can offer more consistency with regard to what developers can expect from their offerings, be confident that this particular book provides a good starting point for those new to REST services.

After discussing basic concepts, design principles, and best practices, the author covers different request response patterns, content negotiation, resource versioning, and response codes, followed by security and traceability, designing for performance, advanced design principles, and emerging standards and the future of REST. The 11-page appendix also offers a glimpse into REST services offered by GitHub, Twitter, and Facebook. One important aspect of this book of which potential readers need to be aware is that it uses the JAX-RS 2.0 API for all code snippets (partial examples) outside of this appendix and chapter 6 (no other frameworks are covered), and probably should not be relied upon even for development using this Java API.

For an introductory look at REST services, the decision to include the appendix was probably misguided, and the chapter on emerging standards and the future of REST, although providing a needed overview of real-time APIs, polling, WebHooks, and WebSockets, is very rushed and should probably be treated by most developers as a bookmark for further research outside what Mehta has to offer. However, the rest of the book (the first 5 chapters) will likely be a good primer for anyone newer to REST services, including those looking to incorporate REST services into other ecosystems such as Spring applications, because it provides a bigger picture view (apart from JAX-RS) than is typically offered by other resources concentrating on specific technologies.

New Book Review: "Building Microservices"

New book review for Building Microservices: Designing Fine-Grained Systems, by Sam Newman, O'Reilly, 2015, reposted here:

As explained in the preface to this book, "microservices" are small, autonomous services that work together, each of which is focused on doing one thing well. While the topic of microservices is fast-moving, the term itself is new even though the idea is not. Be aware at the outset that some other reviewers here did not pay attention to the philosophy used for this book. Newman indicates that, due to the fast pace of change in this space, he tried to focus this book "on ideas more than specific technologies, knowing that implementation details always change faster than the thoughts behind them" and he fully expects that "in a few years from now we'll have learned even more about where microservices fit, and how to use them well".

After covering an introduction to microservices, including some key benefits and downsides, Newman discusses the architect in the context of microservices, followed by modeling microservices, integrating microservices, splitting monolithic systems, designing systems, and scaling microservices, as well as deployment, testing, monitoring, and security. The most heavily weighted chapters are the ones on integrating microservices and scaling microservices, which consume about 32% of the text, followed by the chapters on splitting monolithic systems, deployment, and testing, which together consume about 31% of the text. While I personally would have liked to have seen the development lifecycle chapters at the end of the book, the reader can simply read these last.

The "Microservices and SOA" sidebar of Martin Fowler's "Microservices" article indicates that the "common manifestation of SOA has led some microservice advocates to reject the SOA label entirely" and "others [to] consider microservices to be one form of SOA, perhaps service orientation done right", but "the fact that SOA means such different things means it's valuable to have a term that more crisply defines this architectural style". Newman instead clarifies the microservice approach by explaining it "as a specific approach for SOA in the same way that XP or Scrum are specific approaches for Agile software development", which "has emerged from real-world use, taking our better understanding of systems and architecture to do SOA well".

The author also wisely points out that "microservices are no free lunch or silver bullet, and make for a bad choice as a golden hammer. They have all the associated complexities of distributed systems,and while we have learned a lot about how to manage distributed systems well (which we'll discuss throughout the book), it is still hard. If you're coming from a monolithic system point of view, you'll have to get much better at handling deployment, testing, and monitoring to unlock the benefits we've covered so far. You'll also need to think differently about how you scale your systems and ensure that they are resilient. Don't also be surprised if things like distributed transactions or CAP theorem start giving you headaches, either!"

As a consultant architect, I especially appreciated Chapter 2 ("The Evolutionary Architect"), because of the sections entitled "An Evolutionary Vision for the Architect" and "Governance Through Code", Chapter 4 ("Integration"), because of the summary that the author provides on integration technologies in the context of microservices, Chapter 5 ("Splitting the Monolith"), because it appeals to my recurring project role as data architect, and Chapter 11 ("Microservices at Scale"), because of the summary it provides with regard to what happens when microservice architectures grow from simpler, more humble beginnings to something more complex in the long-term.

That said, I do agree with other reviewers of this book that there is some overlap between the content provided by the author and what is available elsewhere, although my only real qualms relate to the development lifecycle chapters, which probably would have been better placed in appendixes at the back of the text or minimally as concluding chapters. And while the author does point out example by example throughout his discussions, a bulk of these examples are either text descriptions or explained via high-level diagrams, and it would have been helpful if real-world examples were provided at a lower-level technical level of detail, even if, as Newman notes, this space is moving fast.