Erik on Software

Ubuntu on Windows: Part 1

A couple weeks ago, I moved to Windows 10 with the new laptop I received (Dell 5510 32GB RAM), configuring it for dual boot mode with Ubuntu 16.04 LTS.

I was just about to install Cygwin in Windows again, and then remembered a Microsoft announcement from about a year ago.

 

Here's the new landing page for running bash on Ubuntu for Windows:

https://msdn.microsoft.com/commandline/wsl/about

 

If you are interested in using this developer feature, I recommend watching the 16-minute video at the above link. A demo is provided, as well as an architectural explanation of how this all works.

 

In order to turn this feature on, you''ll need to follow the instructions at the following link:

https://msdn.microsoft.com/en-us/commandline/wsl/install_guide

 

The last step takes a bit of time to complete, and the output looks like this:

C:\Users\erik.gfesser>bash
-- Beta feature --
This will install Ubuntu on Windows, distributed by Canonical
and licensed under its terms available here:
https://aka.ms/uowterms
Type "y" to continue: y
Downloading from the Windows Store... 100%
Extracting filesystem, this will take a few minutes...
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: egfesser
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Installation successful!
The environment will start momentarily...
Documentation is available at:  https://aka.ms/wsldocs
egfesser@CENABCD-PRE:/mnt/c/Users/erik.gfesser$

Microsoft is still working to expand the available Linux functionality (what is offered here is a subset), but hey as far as I'm concerned it's worth the install just to get the grep command for searching files.

 

What jogged my memory about the earlier post are some presentations that some folks from Microsoft gave at a Chicago client the other day.

 

Microsoft does seem to be working hard to expand / hang on to its market share. In joking with the folks from Microsoft, they freely admitted that a lot of what is made available in Azure was built using concepts and open source products under the covers that many of us are already familiar with from the Java world.

Chicago Apache Flink Meetup (CHAF):Hands-on Apache Flink Workshop (February 21, 2017)

---

From the promotional materials: 

In this workshop, we will go over a hands on tutorial highlighting building a Flink stream processor.

The workshop will show you how to

1. Set up a stream consisting of Chicago Transit Authority (CTA) bus tracker data

2. Use different types of windowing techniques to generate metrics on the CTA

3. Sink the metrics to a queryable data sink

4. Turn all of the above into a deployable job that can be scaled up.

You will see all of Flink's major components in action: streams, sinks, windows, triggers, watermarks, etc.

A preconfigured virtual machine containing the workshop materials will be available for download prior to the meetup.

Pizza and Soda will be provided!

Joe Olson is a local area developer specializing in applying emerging technology to solve business problems, and is currently focusing on understanding the rapidly developing streaming space.

---

My personal notes:

• Joe has been working with Flink for 1 to 1.5 years now
• the only way to get community going is to get folks involved
• most meetup groups consist of PowerPoint slides
• streaming is hot right now...big data, IoT, and real time ML applying the pressure
• there are currently several emerging frameworks and philosophies that will likely converge at some point
• Flink is where Hadoop was back around 2009
• one way to possibly jump start adoption is through some hackathons
• what we are about to do:
• - ingest CTA bus tracker historical data set
• - use the Flink streaming API to transform this data set into a Kafka topic...this will provide a good overview of the basic streaming model
• - view the data inside Kafka
• - use a separate Flink job to process the data
• - load the data into keyed windows
• - fire the window on count or on timeout
• - aggregate the data when fired
• - sink the aggregated data
• it took Joe a long time to figure all of this out, so he wants to pass all of this on to folks
• the VM is Ubuntu 16.04, Flink 1.2, Kafka 0.9, Kafkatool, IntelliJ 2016.3 Community Version, Scala 2.11, Java 8, Gradle 3.2
• the data consists of one day's worth of CTA data in JSON format
• Joe commented that it is important to know the version of Scala being used when using Flink, otherwise it won't work
• the CTA has a public API that returns minute fixes of all of its buses
• a fix: {"vid": 1958, "tmstmp": "20150211 23:59", "lat": 41.880638122558594, "lon": -87.738800048828125, "hdg": 267, "pid": 949, "rt": "20", "des": "Austin", "pdist": 3429, "tablockid": "N20 -893", "tatripid": 1040830, "zone": "null”}
• just under 1m fixes are created per day...about 250MB uncompressed
• about 2 years of archived daily data also exists: https://github.com/jolson7168/ctaData
• a good deal of Flink jobs fall into this pattern:
• - connect to a data source and set up a stream
• - do processing on each element in the stream
• - agreggrating, filtering, buffering
• sink results of processing (or start another stream)
• at this point, Joe transitioned to demonstrations by first bringing up IntelliJ
• the code is in Github...first do a git pull to refresh the VM, as he made some changes after creating the image
• there are 2 sections of the project...a loader and a processor...the top of the project loads Kafka and the bottom processes the data
• Joe commented that a lot of stuff will get thrown at you in the Flink documentation
• typical steps for a job are to get data from Kafka, process the data, and then sink it
• FixProcessor is a basic job
• all of the build files are set up if you want to compile this into a JAR
• the notes provide the Gradle build command and Flink start command that are needed
• the Flink job manager can be accessed multiple ways, including via browser
• Kafka is strange in that Zookeeper needs to be started separately prior to starting Kafka
• Joe said that we can email or call him with issues, assistance etc with regard to this project
• he noted that everything is installed to the Ubuntu VM home directory
• a graphical way to see contents of Kafka is by using something called Kafka Tool
• create a Kafka topic for output
• there's not a lot you can do with the Flink UI...it's pretty primitive right now
• deploy the JAR to the Flink server via the Flink UI after it is done building
• Boris (one of the CHUG organizers) mentioned that zookeeperutils can be used to simplify
• the JSON file is in the home directory
• the CTA pushes out "half-assed XML files"...the provided JSON file is much better than the XML
• Flink is kind of unique in that it works like SQL in the sense that it determines an optimal plan as to how to run each job
• Joe commented that this is a very trivial job, but it will help you start understanding how Flink processes data
• JSON fields that are not being used are discarded on the way to Kafka
• letting this job run for a couple minutes will get up to 1m records
• to get this job to run faster, "just throw more processors at it"...cluster it
• job #1 takeaways:
• - very simple Flink job
• - connect to source (data file on OS)
• - filter (make sure valid JSON)
• - sink valid fixes to Kafka topic
• - compile into a JAR
• - deploy JAR on server
• - observe execution
• - check out the log files
• Flink logs and you can load into Kibana to search them...don't mess around with the Flink logs directly themselves
• Flink controls the connection to the source and the sink...the logging it's doing is better than anything you'll be able to do
• it will save you a lot of heartache in the long run if you just get used to how Flink logs
• Kafka takeaways
• - start/stop Kafka and Zookeeper
• - observe what is in a Kafka topic
• - use Kafka for both the source and the sink for Flink
• the new version of Kafka apparently replaces offsets with timestamps, which should be more useful
• using Flink 1.2, which just came out...it supports Kafka .10
• state management is better for 1.2
• back to IntelliJ
• Joe then progressed to the next job
• need to get timestamp of each event
• Joe commented that he didn't respect the boundary between event time and process time...when data comes in, you need to tell Flink the timestamp and the format it is in
• once the timestamp is obtained, you have an anchor of when the event took place
• Joe didn't understand the intricacies of events that took place in the past, so didn't get this job done
• there's always 2 boundaries in Flink...the watermark is the latest point in time that Flink knows about
• he he didn't know how to simulate this with historical data
• this job shows how to get the timestamp and how to set the watermark, but doesn't work completely as of yet
• there are 3 different window types and different strategies as to how you can use them
• the windowing feature in Flink is one of it's greatest strengths
• Kafka is starting to go in the same direction as Flink, but Flink has the jump on this by far
• Boris said Kafka went in the opposite, simpler direction because it was too complex
• we don't know who is going to win this contest, but the ideas will remain
• try to follow all of the open source products in this space as to what they're trying to do, and you will get a good idea as to where the next 10 years will go
• someone said that setting up Kafka was the hardest part...Boris argued that it is the easiest part (and I concur with Boris that setting up Kafka is simple)
• the 2 things to take away is that there is a trigger as to when the window fires...you can override this with your own trigger
• it took Joe a long time to figure out how to do this
• there are different ways to handle late data, out of order data...Flink handles these types of cases well
• for example, you could create a 1 hour window, but keep it open a few more minutes to catch any stragglers
• next month's meetup will be grabbing CTA data in real time and applying windows against it
• Flink will give you guarantees
• if you are managing state, you can use RocksDB to...it is optimized for this...RocksDB is the heavy hitter for managing state
• Boris commented that RocksDB is local to the CPU, and it confused the heck out of him...people want to call this state, but it isn't state
• you will run into a lot of trouble if you don't play by the rules that Flink sets...for what it wasn't designed to do...they joked that this is because it's German
• Data Artisans wants to make Flink the database itself...a queriable state feature
• the table is a stream...the stream is a table
• Boris joked that nobody knows what this means, but they all want to do it...Boris said that the stream is not a database, it's a log
• Boris said that his "head spins around" when folks start talking about "infinite streaming" etc
• for debugging purposes, you can just run within IntelliJ using Gradle
• someone asked about unit testing strategies, and he said that they are struggling with this right now
• Boris disagreed...he said that people struggled with Hadoop a few years ago in a similar way
• job #2 takeaways:
• - another simple flink job
• - connect to source (JSON in Kafka)
• - create a fix job from JSON
• - identify the timestamp from the fix
• - key the stream
• - send fixes into windows based on key
• - trigger the window by count or by timeout
• - aggregate items in window on triggering
• - sink aggregate results to PostgreSQL
• - using a custom window
• - using a custom trigger...preserving state
• - using a custom aggregation
• - sinking to a relational database
• - run from within IntelliJ...useful for rapid debugging
• Joe has Flink in production...doesn't know anyone else that does
• Joe encouraged us to present at future meetup sessions, even if it is to present something small for 5 or 10 minutes

Truth Revisited

---

Back in November 2009, during my then annual visit to the RSNA trade show event at McCormick Place in downtown Chicago, I came across a slogan on display in the Cerner Corporation exhibit which read "Single Source of Truth^TM". As a consultant, I was surprised that such a commonly used technology expression was granted a trademark.

And running across this slogan came shortly after my multiple stints at a healthcare insurance firm in which I had recalled the client CIO describing what he needed as "Single Source of Truth". But in revisiting my discussions following the trade show, I was reminded that he had actually described his need as "Single Version of the Truth", and used this expression interchangeably with "Single Source of Data".

In 2010 and 2013, I wrote two variations of the same post to address the differences between "Single Source of Truth", "Single Version of the Truth", and "Single Source of Data". After subsequently coming across the phrase "Source of Truth" within the context of a particular family of data related technologies, I have come to realize that these posts could use a revisit. This post provides a synthesis, starting with some earlier definitions:

Single Source of Data (SSOD) typically indicates that every data element is stored exactly once. In other words, every data element is stored in no more than a single row (record or tuple) of a single database table. Conceptually, this might look similar to the first diagram below. In this diagram, you will notice that although there are multiple databases across the enterprise, the truth that a particular process needs to access can only be found in one location.

Single Version of the Truth (SVOT) typically indicates that multiple versions of the truth can exist within an organization. One of the purposes behind a data warehousing initiative is to resolve these versions of the truth and to provide an “official” version of truth to the organization. One common reason multiple versions of the truth can exist is because such data can reside in organizational silos. Bill Inmon, the “father of data warehousing”, has eloquently described this concept in material he has written over the years.

Conceptually, this looks something like the second diagram below. You will notice that the truth an arbitrary process needs to access can only be found in one location, just as with SSOD, but with SVOT there exists additional logic between this truth and the source databases. This logic resolves any disputes these individual transactional databases might have with each other, and the resolution is stored in the data warehouse for the business process to utilize.


Single Source of Truth (SSOT) is essentially synonymous with SSOD. However, once the term “truth” is referenced, it is usually the case that a data warehouse is involved, because while truth is assumed when a data element can only be found in one place, this assumption does not exist when a data element can be found in multiple places. The distinction that can be made between SSOT and SVOT is that the former can involve linkages to data elements by reference in other locales. When such data elements are updated, propagation is initiated across the enterprise so that duplicate data elements are always updated, resulting in the same “truth” regardless of which element is subsequently read by processes.

Now, as we move further away from the original concept presented, SSOD, there are more variations of methods to achieve truth, but conceptually SSOT looks akin to the third diagram below. As with SVOT, additional logic sits on top of the data in order to resolve differences, but multiple versions of truth do not exist in the original data. In this case, when the truth for a particular data element is updated, logic is initiated to propagate this data to other locales where this data element might also be stored, so that while there is a single source of truth, this truth might exist in multiple places where an arbitrary business process can access it.

(One additional note before moving on to the remainder of this post. As pointed out by an Oracle white paper that was published the year prior to my initial post, multiple versions of the truth might actually make sense for some data elements, because the more a term is connected to the core business, the more definitions of it might exist, with each valid definition being "One Context of the Truth". This post is about tackling the meanings behind several similar-sounding phrases, not tackling the decisions around whether one or more definitions should exist.)

With brevity in mind, one important aspect that my earlier posts did not explain is how the concept of time can come into play. While SSOD indicates that every data element is stored exactly once in the enterprise, this scenario can often mean that each such data element is actually stored once for a given window of time, which means there are no conflicts with regard to version at a particular time. A bitemporal database table does not only store an effective time window for when data within a particular record is valid (i.e. business effective from- and to-dates), but a system effective time window for when the record itself is valid (i.e. system effective from- and to-dates).

So in other words, SSOD may contain multiple versions of a given data element, but only one version exists for a particular time, which means there are no conflicts that would require discussion of SVOT. Data streams handle the collection and delivery of data from continuous events, and one role of these mechanisms is to act as a buffer to hold event data as a protection against interrupted processing as data is ingested by consumers. Event data is not simply consumed and discarded like traditional queue mechanisms, but is instead persisted for a configured duration of time (or other metric). And since messages can reflect the changes of the same data elements within this time period, persistence means that the historical stream of messages can be replayed as if they were taking place for the first time.

During my talk at the September 2016 session of an ongoing Chicago-based internal technology event series held at Centric Consulting, I walked through an illustration of how this works by presenting what change data capture (CDC) is all about, using traditional relational database products as a starting point. Every such database product stores an ordered list of historical database transactions in a log. This transaction log is what is used by the database to guarantee ACID properties in light of failure, reapplying changes as necessary if commits have not yet materialized. A similar concept can be demonstrated when using a database change management product such as Liquibase, which uses an ordered list of historical database schema changes stored in a database change log.

---

---

To round out my presentation of evolving a minimal viable product (MVP) over time, I showed how I extended an existing software application by adding the use of Apache Kafka (a stream processing platform) by way of Spring Integration and a PostgreSQL database extension called Bottled Water (not production ready at presentation time), which can be used to propagate an ordered list of historical data changes. These changes are stored in a Kafka topic as an ordered list conceptually similar to the first two lists discussed above, but it makes sense to mention that one of the differences here is that consumer specific offsets are stored to maintain what each consumer has already consumed (with the given that records can be replayed at a later point).

---

---

The relatively new data streaming community tends to make use of similar-sounding phrases when discussing this topic, most prevalently "Source of Truth" and "Shared Source of Truth". For the purposes of the remainder of this discussion, let's go with the latter (ShSOT). But is this label accurate? Is there any overlap here with "Single Source of Data", "Single Version of the Truth" or "Single Source of Truth"? 

One of the problems with using a single centralized database to hold transactional data for a large distributed system is that it becomes increasingly harder to maintain synchronized state for the current state of the business. With a streaming architecture, data records continuously flow between data sources and applications. The ShSOT is actually stored in shared in-motion event streams, sources of which some have come to call "infinite data sets". These event streams are what store business history, with applications optionally storing local views of this data.

During a recent discussion, my audience reacted with a quizzical response when I commented that Kafka is essentially a database. While it may not have the same functionality as a traditional database, and some might prefer to call it a data store, queue, or "message transport", there is no arguing that Kafka comprises an organized collection of data, and that traditional databases actually consist of event streams since they store ordered lists of historical database transactions. As Martin Kleppmann commented during his talk at Strange Loop 2014, the traditional database is essentially being turned inside out.

---

---

In the above diagram, topic A on the left is populated by an initial service, which is in turn consumed by another service and a stream processor such as Apache Flink, Kafka Streams, or Spark Streaming. Since this topic seeds the data stored in the other downstream topics and local data stores, it is the initial source of data that is shared between the first two consumers. A traditional queue would delete a message that was processed by a consumer, but in this case a message will still be available to other consumers since each such consumer keeps track of messages it has already read. And it is still available to the same consumer if the offset is reset to reflect a past point in time (e.g. in cases where the code for the consumer has been changed, and historical data needs to be replayed by this new code).

So is there any overlap between ShSOT and SSOD, SVOT, or SSOT? In some sense, parallels can be drawn between these concepts. However, traditional databases consist of static data from the standpoint of consumers. In other words, a database does not keep track of records which have been read per se with respect to the order in which records are to be read by consumers. In the above example, topic A can be seen as the SSOD for data elements which are propagated or used as input for additional processing downstream, but does it make sense to refer to SVOT or SSOT?

Probably not. If the same data element is represented with different versions within the same data stream, it is a matter of time until conflicts are resolved, due to elapsed time, although keep in mind that consumers may need to apply additional processing logic unrelated to transpired time. Data streams are often used to consume events which are essentially independent of one another (e.g. in the case of events representing simple forms of state etc), or distinct in nature from those found externally to them, so such conflicts will likely not come into play. In this sense, with the presumption that such data representing the same things is not being consumed from multiple sources, ShSOT is the same as SSOD.

That said, it's probably best to refrain from using similar-sounding phrases across different technology domains. If you do, be prepared to explain. As I wrapped up with my earlier variations of this post, make sure you clarify what is meant by any expressions that you use so that opportunities for miscommunication are minimized.

Google Analytics: 50k Pageview Milestone

According to Google Analytics, pageviews for this blog hit the 50k milestone on February 2, 2017. Below is a geo location plot based on city locations of visitors.

---

---

The following are the top-10 cities of origin:

(1) New York, (2) Chicago, (3) London, (4) Bengaluru, (5) Sydney, (6) Toronto, (7) Melbourne, (8) San Francisco, (9) Paris, and (10) Singapore.

The following are the top-10 countries of origin:

(1) United States, (2) United Kingdom, (3) India, (4) Canada, (5) Germany, (6) France, (7) Australia, (8) Brazil, (9) Netherlands, and (10) Russia.

Workshop Notes (February 1, 2017): IBM dashDB Local MPP Proof of Technology

Hyatt Center

IBM Technical Exploration Center

February 1, 2017
9:00 AM - 5:00 PM

---

---

From the promotional materials:

IBM dashDB Local MPP Proof of Technology

IBM dashDB Local is a next-generation data warehousing and analytics technology for use in private clouds, virtual private clouds and other container-supported infrastructures. It is ideal when you must maintain control over data and applications, yet want cloud-like simplicity.

dashDB Local is part of a family that shares a common SQL engine and other database technologies across different deployments so you can place the right workload on the right platform—and move between them without application change.

It includes in-memory processing for delivering insanely fast answers to queries, as well as MPP for scale out and scale up capabilities as data grows. It also provides scale-in capabilities to free resources after peak workloads complete.

Agenda:

• Setup and Introduction - Introduction to dashDB Local
• GPFS FPO - Create GPFS FPO feature for clustered file system
• dashDB MPP Installation - Installation of Docker containers for dashDB in MPP deployment
• dashDB and Docker Administration - dashDB and Docker administration for day-to-day work
• dashDB Web Console - Use of dashDB web Console for data load, administration of tables, Run SQL
• Query Monitoring - Comprehensive monitoring of Analytics workload

---

---

My key takeaways:

• Vikram Khatri was the presenter. He works for IBM out of North Carolina.
• This workshop was the second stop in a 4 to 5 stop roadshow which started in New York.
• For the workshop, he wrote a detailed (roughly 100 page) tutorial that contains step by step instructions as well as dozens of SQL and Docker scripts which he plans to distribute (the Docker portion is of value for non-related work as well).
• In addition to dashDB, we spent considerable time discussing Docker, since dashDB Local MPP is a Docker based database.
• He was able to answer a bunch of my questions, which was helpful, since I have run across some conflicting information on the web.
• Long story short, dashDB is *not* a hybrid database as in HTAP (hybrid transactional / analytical processing).
• A preemptive decision needs to be made as to whether transactional or analytical processing is needed for a database.
• The two separate database products are dashDB Local for transactions (row-based), and dashDB Local for analytics (column-based), but the former has not yet been released.
• There is also another product called dashDB for cloud. Unlike the other two products, this product is bare metal rather than Docker based, although the plan is to move this to Docker as well.
• He has the same opinion as me that Hadoop is on the decline and will be largely replaced by HTAP etc database products in coming years.
• He noted that this opinion goes against IBM marketing for BigInsights, which is a Hadoop-based product. 
• While walking through the history of database products, he commented that companies have wasted millions of dollars on Hadoop, only to go back to relational databases.

---

My personal notes:

• the workshop description can be found at this link
• presenter was Vikram Khatri from IBM (North Carolina)
• event organizer was Rob Beal from IBM (Chicago)
• apparently about 25 to 30 people registered to attend this event, but less than 10 showed up
• the first 3 of 6 lab exercises (lab01, lab02, lab03) were already performed for clusters beforehand to save time on downloading, installing etc
• at the outset, Vikram commented that individuals with DB2 experience should see some familiarity with what is presented during the workshop
• labs require only basic skills in Linux and databases
• the clustered file system on each workshop machine is visible to all 4 machines/nodes
• most labs will be performed from node dash01, but dash02, dash03, and dash04 can also be used
• most labs will be performed via command line using Gnome terminal and root user...just type 'root'
• at command prompt, # is root and $ is user
• dash01 is the head node...the others are data nodes
• the focus today is dashDB, but IBM has a complete portfolio including Cloudant and BigInsights for Apache Hadoop
• Cloudant is a document store comparable to MongoDB
• "relational databases are not going away...they're going to stay"
• Vikram mentioned prior solutions which tried to fit data into relational databases using BLOBs and CLOBs, but some gaps are best addressed by non-relational databases
• he doesn't see a big market for Apache Hadoop or BigInsights...these are on the decline now
• he thinks that HTAP etc will replace Hadoop
• industry is moving away from gathering different components and fitting them together, to moving to ready-made, self-contained platforms like dashDB
• we grew up in the structured data world
• we started to build on top of these tables for analytical purposes, which led to star schemas, snowflake schemas etc
• and we were doing this on row organized tables, retrofitting for about 15 years
• but this didn't look right...why read the whole system of record?...so column organized tables were devised
• then these databases started to be used for OLTP, which was a complete mistake
• dashDB for Analytics provides ACID compliance guarantees
• companies have wasted millions of dollars on Hadoop, only to go back to relational databases
• "sometimes we make stupid decisions not thinking things through"
• dashDB for Analytics and DashDB for Transactions are apparently two separate products
• I confirmed with Vikram that dashDB should not be considered an HTAP product, since a preemptive decision needs to be made between transactional and analytical purpose
• it was interesting that attendees asked what I meant by "hybrid database"
• this whole movement started with NoSQL, followed by not-only-SQL
• Vikram reiterated that he doesn't see the future in BigInsights / Hadoop...he commented that this is his opinion, not the opinion of IBM
• while these technologies are still relevant, they were applied for the wrong purposes
• some questions from the audience resulted in Vikram saying that we're not trying to force MongoDB document data into DB2
• he reiterated that we've been able to store document data etc in relational databases for years using BLOBs and CLOBs
• but he followed this up by saying that it's going to be the document store which is the best for storing this type of data
• "it took IBM 25 years to figure out how to make money from something that is free"
• IBM made a mistake by pursuing token ring rather than TCP/IP a long time ago...and look where Cisco is now
• I asked him whether the IBM "open data platform" is synonymous with open source, and elaborated by saying that IBM is packaging open source products together
• another attendee clarified by saying that the open data platform is an industry standard to which IBM and Hortonworks et al conform, but not Cloudera et al
• "the new reality here is software as a service"
• someone else asked about Bluemix Data Connect...this is for data integration...Watson is available via this product
• noticing some confusion, Vikram commented that "we have been changing names rapidly"
• "Watson is not a product...it's a platform providing many different things"
• "inside dashDB, it's all DB2"
• if you know DB2, you can use it like DB2, but if you don't it will do things for you by focusing on the data, not administration
• most of the administration is taken away, but there is still some administration to do
• Vikram pointed to me a few times after I mentioned HTAP databases...he pointed to me again and said we will see more and more of these capabilities in the future
• "it remains to be seen which direction the industry goes"
• a new product will be called "dashDB Cloud"
• dashDB Local is a Docker container based technology...dashDB Cloud is deployed to a bare metal server
• dashDB for Transactions is row-based DB2...dashDB for Analytics is column-based DB2
• as of right now, there is a 1:1 relationship between Docker images and machines/nodes...multiple form a cluster (clarified 7/5/2017: see comment)
• only available right now on Redhat Linux 7.2...this workshop uses CentOS 7.2 which is compatible
• data is encrypted at rest...someone asked about DB2 11, so I asked whether he meant data is encrypted by default with dashDB, and said correct...with DB2 it needs to be turned on
• Apple has been using Time Machine for a long time...a "union file system...Docker is a union file system as well
• Docker is a container based technology
• the difference with Docker is that this technology was made "very simple"
• an instance of a Docker image is called a container...a container is an instantiation of an image
• Vikram went into some basics about Docker, and elaborated a bit more when someone joked that they're new to Docker but realizes this isn't a Docker class
• the best example that he gave for Docker image layers is how an image can use a different version of GCC or some other file than the host operating system beneath
• Vikram concurred with me that this is what provides consistency across machines
• dashDB Local requires a clustered file system
• IBM GPFS, GFS2, or VXFS can be used...these are all POSIX compliant clustered file systems
• someone commented that GPFS is expensive, to which Vikram replied "GPFS is not part of the dashDB license...GPFS provides a lot of good things...forget about the price right now"
• GPFS can also be used for traditional DB2
• "you will be amazed at what GPFS can do"
• this workshop uses IBM GPFS with FPO...file placement optimizer
• with GPFS you don't need a SAN
• GPFS provides multiple copies of data across machines
• GPFS provides significantly better performance than HDFS...it's not new, been around 25 years
• dashDB Local for Transactions is not available yet...just dashDB Local for Analytics
• while going over the steps to get access to a Docker image, the speaker commented that it can take up to 24 hours to get a Docker ID via https://cloud.docker.com because a human is involved on the other end
• to get going, you just need to pull Docker images to each node, create the "nodes" file on each that lists IP addresses of each node, and then start up dashDB on the head node
• database names are chosen for you
• when you see "You're almost there" in the tailed log, you can hit Control-C to start the dashDB MPP process on the head node
• after you see "Congratulations! You have successfully deployed dashDB" in the log, you are ready

---

[root@node01 04admin]# ./admin02

========================================================

Start dashDB services from the head node

The command to run : docker exec -it dashDB start

========================================================

*************************************************

Starting services. This might take a while ...

*************************************************

Validating nodes file /mnt/bludata0/nodes ...

#####################################################################

 Check cluster connectivity

#####################################################################

Checking status of node node01

Checking status of node node02

Checking status of node node03

Checking status of node node04

* Network check: PASS

#####################################################################

 Performing cross-node communtication test 

#####################################################################

The dashDB configuration: MPP

==============================================

Running communication test on node: node01

==============================================

Running database port check ...

* port(s) 50000-50001,60000-60024: OPEN

 

Running Spark port check ...

* port(s) 25000-25005: OPEN

 

Running SSH port check ...

nping_event_handler(): READ-PCAP killed: Resource temporarily unavailable

* port(s) 50022: OPEN

 

 

==============================================

Running communication test on node: node02

==============================================

Running database port check ...

* port(s) 50000-50001,60000-60024: OPEN

 

Running Spark port check ...

* port(s) 25000-25005: OPEN

 

Running SSH port check ...

* port(s) 50022: OPEN

 

 

==============================================

Running communication test on node: node03

==============================================

Running database port check ...

* port(s) 50000-50001,60000-60024: OPEN

 

Running Spark port check ...

* port(s) 25000-25005: OPEN

 

Running SSH port check ...

nping_event_handler(): READ-PCAP killed: Resource temporarily unavailable

* port(s) 50022: OPEN

 

 

==============================================

Running communication test on node: node04

==============================================

Running database port check ...

* port(s) 50000-50001,60000-60024: OPEN

 

Running Spark port check ...

* port(s) 25000-25005: OPEN

 

Running SSH port check ...

* port(s) 50022: OPEN

 

 

#####################################################################

Check status of clustered filesystem

#####################################################################

Checking status of the clustered filesystem on node node01

Checking status of the clustered filesystem on node node02

Checking status of the clustered filesystem on node node03

Checking status of the clustered filesystem on node node04

* Clusterfs check: PASSED

Starting services on node node02

Starting services on node node03

Starting services on node node04

Current  total memory:  6692 MB

Previous total memory:  6692 MB

Current number of maximum database partitions per node : 1

Previous number of maximum database partitions per node: 1

Current spark share : 10% of total memory.

Previous spark share: 10% of total memory.

Total memory, maximum database partitions per node and spark share are unchanged.

Run hardware inventory collection

+----------------+------------+----------------+-------------+

|                | # of nodes | # of CPU cores | Memory (MB) |

+----------------+------------+----------------+-------------+

| Current        |     4      |       8        |    26768    |

| Max this month |     4      |       8        |    26768    |

| Max last month |     4      |       8        |    26768    |

| Max all time   |     4      |       8        |    26768    |

+----------------+------------+----------------+-------------+

Redirecting to /bin/systemctl restart  slapd.service

Redirecting to /bin/systemctl status  slapd.service

Redirecting to /bin/systemctl restart  rsyslog.service

Redirecting to /bin/systemctl restart  named.service

Redirecting to /bin/systemctl restart  sssd.service

Redirecting to /bin/systemctl restart  saslauthd.service

Redirecting to /bin/systemctl restart  sshd.service

Redirecting to /bin/systemctl restart  sendmail.service

Redirecting to /bin/systemctl status  rsyslog.service

Redirecting to /bin/systemctl status  named.service

Redirecting to /bin/systemctl status  sssd.service

Redirecting to /bin/systemctl status  saslauthd.service

Redirecting to /bin/systemctl status  sshd.service

Redirecting to /bin/systemctl status  sendmail.service

[18:07:10]  INFO: Starting dashDB instance

 

dsserver_home: /opt/ibm/dsserver

port: 9300

https.port: 8443

status.port: 9302

SERVER STATUS: INACTIVE

......

dsserver_home: /opt/ibm/dsserver

port: 9300

https.port: 8443

status.port: 9302

SERVER STATUS: ACTIVE

Log: /opt/ibm/dsserver/logs/dsserver.0.log

Running dashDB Health check ...

###########################################################################

#####           --- dashDB stack service status summary ---           #####

###########################################################################

Redirecting to /bin/systemctl status  slapd.service

 

 

SUMMARY

 

LDAPrunning                   : SUCCESS

dashDBtablesOnline            : SUCCESS

WebConsole                    : SUCCESS

dashDBconnectivity            : SUCCESS

dashDBrunning                 : SUCCESS

 

###########################################################################

#####             --- dashDB high availability status ---             #####

###########################################################################

 

Configuring dashDB high availability ...

Stopping the system

Stopping datanode node02

Stopping datanode node03

Stopping datanode node04

Stopping headnode node01

Running sm on head node node01 ..

Running sm on data node node02 ..

Running sm on data node node03 ..

Running sm on data node node04 ..

Attempting to activate previously failed nodes, if any ...

SM is RUNNING on headnode node01 (ACTIVE)

SM is RUNNING on datanode node02 (ACTIVE)

SM is RUNNING on datanode node03 (ACTIVE)

SM is RUNNING on datanode node04 (ACTIVE)

Overall status : RUNNING

###########################################################################

***********************************************************

****           Successfully started dashDB             ****

***********************************************************

*

* Next steps:

*

* 1. Please use Ctrl+c key-sequence to detach from the Docker logs console.

 

* 2. Log in to the web console using the following information:

*        URL: https://192.168.142.101:8443

* 3. If you've forgotten the bluadmin password, reset it

*         by running "docker exec -it dashDB setpass <new password>". Other user management

*         should be done via the web console.

*

***********************************************************

---

• the workshop slides mention version 1.4.1, but Vikram mentioned that he needed to work late the prior night to move to version 1.5.0 to all of the machines in the lab due to the recent release
• his response to my asking how to upgrade was that he would cover this topic later, but he also commented that it is "much easier" to do so than for DB2
• the version of the software stack is located at /mnt/clusterfs/SystemConfig/'hostname -s'/.build_version
• as we breaked for lunch we started lab04 - dashDB and Docker Administration
• during lab04, I asked Vikram whether all 4 nodes were running on my physical machine, and he said yes...the machine had 32GB RAM and each node was allocated 7GB RAM
• at one point I needed to restart the head node because I had run some commands to add a node that were not supposed to be run
• in order to get dashDB running again, I needed to issue the following commands: 'docker exec dashDB stop' followed by 'docker exec dashDB start'
• working through lab05 - dashDB Web Console was next
• Vikram told me that he is based in North Carolina...this is the second time he has presented his materials...a couple more stops on his roadshow...New York was first
• you have complete control over the Docker images...a container is an instance of an image
• you have the key to the front door of the Docker container...the following provides an interactive shell within the container: 'docker exec -it dashDB bash'
• in a cloud environment you cannot do this
• remember, Docker became popular because of the union file system
• virtual machines are much more expensive in terms of resources than Docker...Docker is much more efficient than virtual machines
• virtual machines are not going to go away, just like COBOL is not going to go away, but growth in terms of adoption will disappear
• right now, IBM is releasing on a monthly basis
• Vikram reminded IBM database administrators in the room that learning Docker and dashDB are some good steps to reinvent themselves
• he commented that if our organizations do not have a Docker repository yet, they are behind...and "laggards will suffer"
• "dashDB exceeded my expectations"
• "financial customers typically want to be cutting edge"
• "retailers are the last...they don't want to do anything unless they need to"
• dashDB Local was released about 6 months ago
• I asked Vikram twice if he can give us a sense of how many customers exist...and then the ratio between DB2 and dashDB customers...but he didn't delineate these figures
• "I was late into the game adopting Docker myself...that was my ignorance"
• to check the resources that are running: 'docker top dashDB'
• NGINX is a small print web server that is often used to route to Docker containers
• the big difference between virtualization and Docker is that Docker uses the kernel from the host
• dashDB Cloud is also moving to containers...right now it is bare metal...if you want to increase nodes, for example, IBM staff currently does so for you
• the db_migrate command can be used to transfer data from Netezza
• minor upgrades are easy...in response to my questions about upgrade risk, Vikram said that by the time major upgrades become available IBM will have a way to do major upgrades
• things you typically do via a SAN administrator can be done yourself via GPFS
• while working through lab05, I ran into an issue performing step 5.10..."unable to find user root: no matching entries in passwd file"...Vikram already opened up a ticket with Docker...the temporary fix is to run this command from another node
• I finished lab04 and lab05, but didn't complete lab06 ("dashDB query monitoring")