Erik on Software

Personal Reading List: 4Q2020

My personal reading list for October, November, and December 2020.

Color Key: Special Notes, Completed Reading. Updated throughout the Quarter.

---

At Bat (Schnell)

---

The Model Thinker: What You Need to Know to Make Data Work for You, by Scott E. Page, Basic Books, 2018.

The Software Architect Elevator: Redefining the Architect's Role in the Digital Enterprise, by Gregor Hohpe, O'Reilly, 2020. Copy provided by Amazon.

---

 At Bat (Langsam)

---

Fluent Python: Clear, Concise, and Effective Programming, by Luciano Ramalho, O'Reilly, 2015.

Learning Spark (Second Edition), by Jules S. Damji, Brooke Wenig, Tathagata Das, and Denny Lee, O'Reilly, 2020. Copy provided by Databricks.

---

 On Deck

---

Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith, by Sam Newman, O'Reilly, 2020. Copy provided by Nginx.

Practical Statistics for Data Scientists: 50 Essential Concepts, by Peter Bruce and Andrew Bruce, O'Reilly, 2017. Copy provided by MarkLogic.

Python Programming for Beginners: A Kid's Guide to Coding Fundamentals, by Patricia Foster, Rockridge Press, 2020. Copy provided by Amazon.

---

In the Hole

---

Presto: The Definitive Guide, by Matt Fuller, Manfred Moser, and Martin Traverso, O'Reilly, 2020. Copy provided by Starburst.

Community Comment: Part 2

The comments I provided in reaction to a community discussion thread on September 21, 2020:
https://www.linkedin.com/feed/update/urn:li:ugcPost:6712112282559758336?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A6712112282559758336%2C6712156655460601856%29&replyUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A6712112282559758336%2C6713928878362025984%29

Director of Data Analytics and Application Development: Some best practices for implementing cloud data lake:
Cloud Data Lake Best Practices
https://www.linkedin.com/pulse/cloud-data-lake-best-practices-jasmeet-singh/

Senior Solutions Architect at Databricks: Its interesting that you recommend using parquet on the data lake, but delta lake only if needed. How would you perform the merge operation on parquet between layers if you don't utilize delta lake? 

Director of Data Analytics and Application Development: That is a very good question, my understanding is it depends on the architecture. Lets say data bricks or spark is not part of your architecture you can just load parquet files into time partitioned folders and over write those files. You can then create external tables in Synapse DW to serve. Delta lake has its own benefits like time travel so in some scenarios it can compliment DW layer.

Senior Solutions Architect at Databricks: Time Travel is a nice feature of Delta Lake, but some of the more beneficial features are around the management of data with common ETL patterns. The ability to Insert, Update, Delete, and Merge with simple SQL commands on cloud object storage is an amazing value add of Delta Lake. Without Delta Lake these types of operations on object storage or in data warehouse layers would be very expensive in the cloud, no? I've seen the most successful patterns emerge using Delta Lake on cheap cloud object storage to do ETL, and using a cloud data warehouse as a serving layer for use cases that require that high SLA, and can absorb the higher cost.

Director of Data Analytics and Application Development: Yes I agree but not all organizations need to use Delta Lake as serving layer, for many MPP DW with data lake storage on back end will be sufficient, you can have all the ACID compliance, insert , merge , update in DW Layer. Please check how azure synapse is doing it, they are using azure storage not built in storage so you still get benefits of lower cost storage but do not have to maintain both delta lake and DW. "With decoupled storage and compute, when using Synapse SQL pool one can: Independently size compute power irrespective of your storage needs. Grow or shrink compute power, within a SQL pool (data warehouse), without moving data. Pause compute capacity while leaving data intact, so you only pay for storage. Resume compute capacity during operational hours" Just to clarify I am not against Delta Lake I am trying to say please evaluate if you need it and in some cases it will have additional value.

Gfesser: So for best practice #3, I'm trying to reconcile your comments here with the following statement: "Implement Delta Lake only if needed, If you already have separate data warehouse layer as part of your architecture then use that instead". At a minimum, I suggest stating that Delta Lake is recommended if Spark (the minimally compatible version) is already being used, because this is the architectural dependency. Alternatively, multiple scenarios could be laid out, but since the purpose here is to explain best practices, perhaps you could just state whether you consider use of Spark a best practice. I don't think use of a "separate data warehouse layer" explains whether Delta Lake should be implemented.

Community Comment: Part 1

The comments I provided in reaction to a community discussion thread on October 13, 2020:
https://www.linkedin.com/feed/update/urn:li:activity:6721405361934479360?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6721405361934479360%2C6721451805441622016%29

Software Engineer at Facebook: A lot of developers I talk to have trouble understanding database transactions and why they're incredibly useful.

A transaction is a way to group several reads and writes into ONE logical unit. When we encapsulate reads/writes into transactions, we can (usually) assume several guarantees, described by "ACID".

Atomicity
- either ALL the reads/writes succeed or ALL fail. If a fault occurs after some of the writes, then the database undos them and aborts.
- error handling is MUCH easier, we don't have to worry about some writes succeeding and some failing.

Consistency
- a transaction cannot violate invariants (or constraints) of your database.
- Ex. a constraint limits an integer attribute to values between 1 to 10. Transactions cannot violate that.

Isolation
- Most databases are accessed by several clients at the same time, this can cause concurrency problems.
- Databases ensure that when all transactions are committed, the result is the same as if they had run serially (one after another).

Durability
- Once a transaction is committed, you won't lose it!

Not ALL databases have ACID transactions! Some may make the tradeoff for speed or accessibility.

Gfesser: Definitely important for developers to understand. Just keep in mind that database products tend to implement ACID differently, so developers cannot simply rely on claims of compliance. Also, keep in mind that "C" arguably doesn't really belong in the acronym. Kleppmann's summary perhaps said it best, commenting that it was "tossed in to make the acronym work" in the original paper. The reality is that "atomicity, isolation, and durability are properties of the database, whereas consistency (in the ACID sense) is a property of the application. The application may rely on the database's atomicity and isolation properties in order to achieve consistency, but it's not up to the database alone."
Designing Data-Intensive Applications
https://www.erikgfesser.com/weblog/2018/06/new-book-review-designing-data-intensive-applications-big-ideas-behind-reliable-scalable-maintainable-systems-martin-kleppman.html

Media Query Source: Part 30 DZone (US digital magazine)Guide to databases: the backbone of softwareExecutive insights on the state of databasesOne of 19 contributors to 9 key findings

---

My responses ended up being included in a publication at DZone (July 9, 2019) for an article entitled "Executive Insights on the State of Databases" (pages 32-33). Above image from cited publication.

---

(No transcript available for the verbal responses I provided to a media outlet on April 22, 2019.)

---

See all of my responses to media queries here.

Media Query Source: Part 29 CMSWire (US digital magazine)Digital transformation change management"Digital transformation" needs to be definedPlanning essential, but plans should be agile

---

My responses ended up being included in an article at CMSWire (May 11, 2018) entitled "Change Management for Digital Transformation". Above image from cited article.

---

(No transcript available for the verbal responses I provided to a media outlet on May 2, 2018.)

---

See all of my responses to media queries here.

Building a Data Platform on AWS from Scratch: Part 2

---

This article is the second in a multi-part case study series discussing the data platform my team and I built on AWS for the CIO of a Chicago-based global consultancy. As explained in the first part, one goal for this platform was to centralize data assets and data analysis across the firm using a common architecture, building on top of it to meet the use case needs of each organization. Additionally, business units and consultancy practices had become siloed, each making use of disparate processes and tooling, so in addition to centralizing data assets and data analysis, another goal was to implement the concept of data ownership, enabling the sharing of data across organizations in a secure, consistent manner.

The conceptual architecture comprising data pipelines, data lake, data models, visualizations, and machine learning models is laid out in this previous piece. Alongside this discussion, key guidelines from the CIO are considered, including organization by something called an "insight zone", as well as industry terms such as bronze, silver, and gold data as these relate to the data pipeline stages built for the platform. Finally, a time-sensitive look at the tech stack selection process is provided, alongside the technologies we ended up implementing for the platform minimum viable product (MVP). So in order to get the most out of this article, it might make sense to read through these discussions first.

Rather than walking through details of the implementation architecture, the goal of this second article is to walk through some architectural discussion points relating to what was implemented for the remaining build effort. These points are presented here in somewhat of a piecemeal fashion, albeit with ties back to the first article, by elaborating on some of the initial architectural decisions that were made, how these decisions held up during the course of the project, and aspects of the build-out one might want to consider as they tackle their own data platform challenges. As before, in order to cater this article to a more diverse audience an attempt has been made here to explain from the context of both data and software development work, as many traditional practitioners in these areas do not have significant experience in both.

The data platform

Before moving forward, it probably makes sense to discuss the "data platform" in the context of other similarly sounding terms such as "data mesh", and what distinguishes a data platform from an "application platform". Unfortunately, there exist no universal definitions for these terms, just like many other concepts in technology and business. However, it so happens that, unbeknownst to us at the time, an article by martinfowler.com guest author Zhamak Dehghani entitled "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh" was published (May 2019) just after this project started. In the context of this article, what we built was somewhat of a hybrid between a third (current) generation data platform and a fourth generation data platform, summarized by the following table.

Generation	Characteristics	Notes
First	proprietary EDW and BI platforms high price tags significant tech debt, low positive business impact	tech debt can often include thousands of unmaintainable, infrequently used jobs, tables and reports
Second	complex big data ecosystem traditional data lake as a silver bullet batch jobs operated by central team of data engineers over-promised, underrealized	at best, has enabled pockets of R&D analytics
Third (current)	more or less similar to second generation unified batch and stream processing using frameworks fully embraced cloud based managed services	addresses some gaps of previous generations, such as real-time data analytics reduces costs of managing big data infrastructure
Fourth	common data infrastructure as a platform convergence of the following: - distributed domain driven architecture - self-serve platform design - product thinking with data domains owned by independent cross-functional teams under centralized governance, enabled by shared self-serve data infrastructure	independent cross-functional teams have both embedded data engineers and data product owners far from a landscape of fragmented silos of inaccessible data

Other than the decision we made with our client to not focus on real-time data analytics, a comparison between what these two data platform generations provide and what we built would show that almost every other characteristic was met. One key difference, however, is that we chose to also centralize domain-specific data processing on the "common data infrastructure as a platform" described by the author. To be clear, what we built did not follow the practice of using a "central team of data engineers" for domain data like second generation data platforms. Instead, this infrastructure was laid out using secure, domain-specific insight zones, each owned by independent cross-functional teams and managed by a product backlog.

In contrast, when considering what are commonly referred to as application platforms, one soon finds that the data layer in such solutions is often only considered after the fact, or not at all. And this often proves to be a huge barrier to progress, especially when working with preexisting data, in which case the data layer should be tackled first. Generally speaking, it is typically much easier to rewrite software when needed, whereas with the data layer this is not the case. As mentioned in the first part of this series, one guiding principle we provided to our client is that everything should be versioned. One needs to keep in mind, however, that software can simply be replaced with new versions of code, whereas with the data layer this may not be the case due to the state that is represented by currently stored data.

Methods to address schema evolution exist, but at some downstream consumption point these changes will need to be handled. Application platforms often make use of events in JSON format, for example, and while these permit less stringent data structures through schema-on-read rather than schema-on-write, versioning still needs to be performed for these data structures, as do the APIs created to consume these events. Whereas with application platforms the focus is typically on narrow definitions of output, whether these be for human users or other applications, with data platforms the output can be extremely broad, covering many possibilities because the focus is on the data itself, which will likely serve a variety of use cases.

Platform boundaries

We chose to make use of a central, core team to build and manage the infrastructure along with all the common, configurable components deployed to this infrastructure. Because of this, we needed to define platform boundaries. We knew we needed to determine scope of what was to be considered "on" and "off" the platform, because not only did we expect many sources of data to eventually feed reporting, visualizations, and machine learning models, we were also working within a time box to build the minimum viable product (MVP). As such, we needed to limit team responsibilities in order to release a working platform to production on time, providing governance for a reasonable scope, so we decided that visualizations and processes to send data from source systems to the initial ingress S3 bucket were to be off the platform from the context of the project (the equivalent to the platform at that point in time).

After stakeholders carefully consider platform boundaries over time, these may expand, but there are many aspects to consider, such as the development team resources needed to support the platform. Within the context of this project, however, these boundaries remained the same for the duration of the project. One challenge we had shortly following release of the MVP was that the client decided to quadruple the number of resources working on the platform, during a time period when we were still working to finish implementation of DevOps tooling and processes, and we came to realize that, while well understood platform boundaries existed, the client had no enterprise architecture team to help govern what was off the platform.

A hybrid 3rd & 4th generation data platform

The first part of this case study depicts the platform boundary situated between the ingress and landing S3 buckets, as well as the security we configured to demarcate inter-insight zone security and intra-insight zone security, all simplified to focus on data store components. The aforementioned article provides a diagram depicting the "data mesh architecture from 30,000 foot view", which includes other aspects beyond the data being processed. The following Venn diagrams are intended to provide an apples-to-applies comparison of these two.

Worth noting are two key differences. The first is that a centralized infrastructure is used for everything in the data platform that we built, other than the infrastructure needed to send data from source systems to the initial ingress S3 bucket. The second is that we decided to distinguish between components on and off the data platform, each of which are subject to different governance and standards. While some firms are liberal with respect to how domain teams tackle the serving of data, we recommended that an enterprise architecture team govern what was to be built off the platform. As our client did not have such a team, we recommended that this was a great time to establish one.

As an example of why such a team is important, we discovered that one insight zone development team started using a data tool with which none of us was familiar. This tool also happened to be the free version of a commercial offering, potentially exasperating the issue because use of it may end up leading to additional costs down the road. The development team started using this tool because they claimed the API provided by Salesforce was problematic, not permitting them to ingest data from it using Apache Spark, and that this tool provided resolution to this issue. We immediately realized that if the non-Salesforce tool they began using provided a solution, this tool must be using the API provided by Salesforce under the covers, and so there must be another explanation for the issue.

A demo by the development team showed that while they were indeed getting an error when using the API provided by Salesforce, it turned out that some of the aggregate data fields they were trying to ingest were the root cause, and since the original data fields at the individual (or disaggregated) level were available and what were needed for the platform, these aggregate data fields could simply be excluded from ingestion. When presented with this solution, the development team indicated that they could not make use of it because the business requirements presented to them indicated that all fields need to be ingested from certain Salesforce objects. In turn, we reminded them that business requirements sometimes need to be challenged.

At the opposite side of the platform, the business wanted to make use of Microsoft Power BI rather than Amazon QuickSight for visualizations, due to its feature set and their in-house familiarity with it. That said, we discovered that one insight zone development team was performing a significant amount of data modeling within Power BI. As we explained to our client, while products such as Power BI can be used for data virtualization, which consists of locally implemented data models alongside cached source data reflecting different models, doing so tends to set a poor precedent unless solely used for initial POC work.

For example, doing so prevents visibility for a broader audience, does not enable the ad hoc querying of such data from outside the chosen visualization and reporting product, and leads to inconsistency across locally implemented data models for the same tables. We recommended that when client stakeholders make decisions around use of a visualization and reporting product, they should make sure that configuration of queries and data models can be incrementally versioned alongside the rest of the code base in order to enable effective DevOps pipelines. Doing so becomes especially important for any domain data that is shared with other domains, to help ensure that use of this data is consistent across consumers.

Data stores

While the first part of this case study does not discuss any of the AWS services used for the six different data stages, suffice it to say that half of these use Amazon S3 regardless of the case study that is to make use of the data stored in each of these. The reason this is the case is because all data is to arrive as files, with the intention to progress these files to a state that provides clean, cataloged data. Once a given file is made available in the staging S3 bucket, it can then be queried via Amazon Athena, albeit still stored in what is essentially the same record layout as the original raw files. While storing files in each intermediary state consumes more storage, costs to do so are low and doing so permits the ability of insight zones to compare raw files with transformed versions of these files, which together form data lineage.

However, beyond this exploratory usage is the need to execute performant queries, and one of the driving factors in the selection process to choose a data store is performance. A good sense of the data to be stored and the queries to be performed help assess which data stores might best be used. In other words, expected use cases. Now, this does not mean that all details need to be understood at the outset, but this does mean that it will help determine the best direction to take. At the beginning of this project, in contrast, our client had requested that AWS implement a couple common patterns that would handle all of their data, without knowing anything about it, and this request was obviously a nonstarter.

One thing that should be emphasized that was only briefly mentioned at the conclusion of the bronze, silver, and gold data discussion in the first part of this case study is that gold data is not always intended to be used for analytical purposes, even though some vendors seem to imply that this is the case, especially when it comes to machine learning. One insight zone development team, for example, wanted to go straight to an analytical model because they were initially solely focused on specific visualizations and reporting, and skipping the creation of a canonical model for their domain data would have permitted them to close out some of the stories on the board more quickly. 

However, what if business users wanted to query data structures, stored with source data granularity, with relatable business terms? What if another insight zone wanted this insight zone to share one or more domain objects? What if other visualizations and reporting needed to be developed for this same data? 

As one of the only two client guidelines stated preference to make use of native AWS services, it does not take long to realize that we were essentially limited to three data store options, because none of the others made sense: Amazon S3 + Amazon Athena (Presto), Amazon Aurora, and Amazon Redshift. While there were many factors we took into account, the following table is a simplified version of the scenarios in which we indicated each should be used for this client.

AWS Service	When to Use for Analytical Store?	Notes
Amazon S3 + Amazon Athena (Presto)	Considered the default for the long term Relatively high data volume & moderate query activity Less complex queries since sorting on single column per table, but partitions can also be created Referential integrity between tables not needed (relationships need to be enforced programmatically) Joins between tables of very high data volume not needed For best performance, tables should be 128 MB or more in data volume Since tables created apart from storage, files need to be preemptively optimized	AWS Lake Formation expected to bring row level security via Athena with an upcoming feature With the given that use case needs are met, this option is generally the least costly
Amazon Aurora	Considered the default for temporary MVP use Relatively low data volume & query activity Total data volume less than 64 TB
Amazon Redshift	Very high data volume & query activity More complex queries since multiple sort keys permitted More extensive SQL support, traditional database objects needed Data joins across S3 and Redshift tables via Redshift Spectrum Referential integrity between tables not needed (relationships need to be enforced programmatically)	As of November 2019, the ability to issue federated queries across multiple AWS services is also available, but tests would need to be performed to determine whether this should really be done. With the given that use case needs are met, this option is generally the most costly

Of these options, note that this first option is not technically a database because Athena queries can be executed against data where it is already stored in S3. In other words, the data does not need to be first loaded into another product. However, Athena needs to be told what this data looks like in order to interpret it. In addition, keep in mind that Athena makes use of open source Presto under the covers, making it a distributed product. Redshift is also a distributed product, but since Redshift involves a load step the distribution style can be chosen after the fact. When using Athena, data needs to be preemptively stored as needed by queries. However, while both of these products have nodes, these are largely transparent to the Athena user. Aurora also has nodes, but processing is not distributed.

AWS Glue, AWS Lake Formation, and Databricks

Of all the tooling we considered and ended up implementing for this client, perhaps the greatest discussions are to be had about AWS Glue, AWS Lake Formation, and Databricks, since the functionality of these overlap with one another, and these are moving along the fastest. Of these three, AWS Lake Formation is the least mature, trailing AWS Glue by a couple years, with Databricks looking back at these products from a distance. As discussed in the first part of this case study, Databricks Unified Analytics Platform was not adopted for a number of reasons, including the facts that it is not a native AWS service, and AWS plans to increasingly roll out overlapping features over time, but that it may make sense to include Databricks products at a later point. Let's take a closer look at this timeline.

Date	Event	Notes
August 2017	AWS makes Glue generally available (GA)
December 2017	Apache Spark 2.2.1 released
April 2019	Our client project starts
April 2019	Databricks open sources Delta Lake (version 0.1.0)	Delta Lake 0.1.0 requires Apache Spark 2.4.x
May 2019	Apache Spark 2.4.3 released
June 2019	Databricks adds support for Apache Spark 2.4.3
July 2019	AWS adds Glue support for Apache Spark 2.4.3 and Python 3	AWS Glue prior support solely for Apache Spark 2.2.1 and Python 2.7 (we subsequently standardized on Python 3.6 for AWS Lambda functions, Glue Python shell scripts, Glue Spark jobs, and Glue Development endpoint SageMaker notebooks)
August 2019	AWS makes Lake Formation generally available (GA)	AWS warned us not to use Lake Formation for production, either prior to this milestone, or for an unspecified period of time post-GA, due to expected instability
September 2019	Our data platform MVP was released to production
October 2019	Our meeting with Glue & Lake Formation product manager	AWS presented to us the Glue & Lake Formation roadmap for 4Q2019 through 3Q2020 (under NDA) that was broken down into using, administering, and building data lakes, as well as data lake infrastructure
October 2019	Our project team resources working on data platform quadruple
October 2019	Our data platform core team focus shifts to resolving technical debt and completing DevOps for use by expanded team
January 2020	Planned desupport of Python 2.7 by Python Software Foundation...
February 2020	Our client project ends
April 2020	...J/K, Python Software Foundation decided to delay desupport of Python 2.7 three more months
April 2020	AWS adds Glue support for Spark Structured Streaming	One of the Glue features I had especially been waiting for, even though our focus on this project was not real-time data analytics (I've since tested this feature with Amazon Kinesis Data Streams, and requested that AWS update the Glue pricing page to include this third Glue option)
June 2020	Apache Spark 3.0.0 released
June 2020	Delta Lake 0.7.0 released	Delta Lake 0.7.0 and above requires Apache Spark 3.0.0  (note the existence of relatively immature Delta Lake competitors Apache Hudi and Apache Iceberg, with Apache Hudi officially supported by Amazon EMR, possibly predicting subsequent support by Glue)
June 2020	Databricks adds support for Apache Spark 3.0.0

We explained during our final knowledge transfer sessions to our client in 1Q2020 that architects will need to keep up with AWS Glue releases, Apache Spark releases, and any other releases of supporting products, including key open source libraries being used, as the client continues to build out its product backlog for the data platform. While this case study cannot go into detail about the Glue and Lake Formation roadmap that AWS presented to us during this project, since it is under NDA, it is especially important for client architects to keep up with new features released for these products.

Closing comments

When considering a data platform buildout for your firm, an important thing to keep in mind is that availability of infrastructure isn't synonymous with data architecture, much like it isn't synonymous with application architecture. While infrastructure in many respects will provide the base of a data platform, and the ability to take advantage of DevOps to build out this infrastructure is extremely important, expect that further custom code will likely still need to be written due to the data itself, and how this data is to be used. With respect to the data platform discussed here, much of this code is constrained to the downstream processing involved with any models (data structures and machine learning) that need to be implemented. While upstream processing is genericized and made configurable largely via key/value pairs, downstream processing takes into account actual use cases.

The aforementioned first generation data platform (still being sold in the marketplace) may not require much code, but the tech debt created by these can be extremely high. Tech debt can materialize in many forms, and with such first generation data platforms this often comes about due to overreliance on proprietary tooling. Recent generations can provide much more flexibility. The data platform that we built for this client won out over another system, being built out by another team using a proprietary product, because of the significantly decreased costs provided by our largely serverless architecture that took advantage of popular open source libraries and components in AWS.

Media Query Source: Part 28 TechTarget (US digital magazine)Data streaming hurdles & best practicesConcepts challenging for traditional data prosGreenfield use cases relatively less complex

---

My responses ended up being included in an article at TechTarget (November 25, 2020). Extent of verbatim quote highlighted in orange, paraphrased quote highlighted in gray. Above image from cited article.

---

The responses I provided to a media outlet on October 7, 2020:

Media: I am working on a story looking at some of the hurdles organization run into when deploying big data streaming applications.

 

Media: In what significant ways have real time streaming platforms evolved over the last couple of years for providing the proper foundation for real-time analytics initiatives?

 

Gfesser: Recent real-time streaming products largely follow the same pattern, permitting ingestion of a high volume of data and the querying of this data via varying "windows" prior to being ingested by downstream processes, but there are a considerable number of competing products in this space, with commercial variations typically borrowing either open source code under the covers, or ideas brought about by open source projects. 

 

A brief side note about "real-time" processing. The vast majority of such processing is actually "soft" real-time, typically intended to mean the opposite of traditional batch processing in which data is processed in bulk, typically on a scheduled basis. I have also developed "hard" real-time software that involved processing data within strict time constraints for activity taking place in physical space. Unlike hard real-time processing, the soft real-time processing that enterprises typically need is more graceful because one of the features it increasingly provides is the ability to replay past events. Because hard real-time processing typically reacts to physical events, losing opportunities to react is probable if not carried out in a timely manner. While soft real-time events such as actions taken by users in a web application can also require timely reaction, and advantages exist in preventing such loss, the downstream loss of an upsell opportunity is likely to be considered less detrimental than worker safety in a factory.

 

In addition to permitting ingestion of a high volume of data, the querying of this data via varying windows prior to being ingested by downstream processes, and the replaying of such events, real time streaming products also typically store data for some period of time in order that events can be revisited. The time that events can be stored varies greatly. For example, Amazon Kinesis Data Streams provides the option to store data for an extended period of time of 7 days rather than the default 24 hours, but Apache Kafka provides the option to retain data for an unlimited amount of time. However, retention of data is only one factor to consider of many, as costs to store and read data also need to be taken into account, as is choosing tooling most likely to attract technology talent.

 

In-between the extremes of batch and real-time processing is what is often referred to as mini-batch, and the community has discussed at length over the last several years what this means. Essentially, rather than processing individual events one at a time, such products process events in small batches. Because costs can sometimes be cost prohibitive for individual event processing, and timeliness of such processing may not be extremely critical, there can be benefits in making use of mini-batch. When considering batch versus real-time processing, I typically discuss in the context of scheduled versus unscheduled processing. Can processing wait? What advantages might arise when not waiting and vice versa?

 

Media: What are some of the top hurdles enterprises have to overcome when deploying and managing real time data streaming platforms? For each one, what are some of the best practices for addressing these? 

 

Gfesser: The data processed by real-time streaming products is typically viewed as a stream of events. In the case of products such as Apache Kafka and Amazon Kinesis Data Streams, streams can be processed similarly to database tables, albeit the data in such tables is defined in the context of windows based on varying criteria. For example, a use case only considering data within a certain time constraint will define such tables differently than a use case that permits consideration of straggling events. Understanding these concepts can be challenging for some traditional data practitioners. 

 

Additionally, the very concept of an "event" might take some development teams time to digest, especially with respect to how events might be defined for scenarios currently processed by legacy applications. And from the perspective of managing streams, greater capabilities and flexibility typically means more development. While the capabilities and flexibility that Amazon Kinesis Data Streams and Apache Kafka offer exceed that of Amazon Kinesis Data Firehose, for example, Firehose also requires significantly less management. 

 

Note that deployment of a real-time streaming product in a greenfield scenario will be relatively less complex. When legacy applications are involved, care must be taken to sufficiently analyze existing processes to define events that these create. When defined effectively, a real-time streaming product can serve as the backbone of an entire enterprise ecosystem, providing a single source from which to draw data, especially for cases in which microservice architectures are being built. In such cases, it is prudent to tackle product selection early because it will serve as a central component. Soon afterward, make sure ongoing build efforts address ongoing management, as some teams only test with the easy path in mind and so do not end up being equipped for the longer term that often involves more data or activity.

---

See all of my responses to media queries here.