Erik on Software

Pro Bono: IEEE Chicago Section Statistician - Part 1

Over three years have past since my last post here on my pro bono efforts outside of the workplace. My latest pro bono contributions over the last few months have been working as Statistician for the IEEE Chicago section. The Statistician role is new for the section, and works within the Membership Development team.

Much of my time to date has been spent working to get familiar with the data that is currently available on our membership. The SAMIEEE database uses Oracle Business Intelligence Enterprise Edition (OBIEE) 11g. Unfortunately, my early efforts to obtain programmatic access to the database via IP address and port was not successful, so I create rough data sets via the web front end that OBIEE provides and then export for analyses.

At a high level, my CSV exports from OBIEE are imported to PostgreSQL, and then I use the R language with a package called sqldf to perform analyses. Traditional SQL statements can be run against database tables in this manner as though they were R language data frames, making it much easier to work with data for someone such as myself who is adept at SQL as a technical architect.

My plan is to get into more detail regarding the specifics of my setup in later posts. As part of this first post, I want to share the first visualization I created for the IEEE Chicago section a few months ago. A heat map showing the top 26 technical interest profile subjects (TIPS) of active IEEE Chicago section members, created using the above technologies along with specialized R language functions such as heatmap().

The Membership Development Chair has since used this visualization as part of his marketing efforts associated with trade shows, as part of his presentation on our membership. Creating this heat map was fairly straightforward. Although working with the R language environment and language is not intuitive for someone who specializes in Java, getting the right data to create this visualization was really the hard part.

While the TIPS list might not need an explanation, since it is essentially a list of technical interests in which members have shown interest, notice the labels for the three columns. Data was queried and sorted by primary interest, which indicates whether a member actually performs regular work in the area, followed by some work in the area, and some degree of interest, the least significant category of the three.

Traditional engineering concerns may comprise the bulk of the top 26 technical interest profile subjects (a somewhat arbitrary number that I chose, although the break off as to whether a subject made it to the heat map was set at 50 members in the regularly work category), but I found it very interesting because of my profession that Software Engineering was the top regular work subject, and one of the top subjects in the some work category. And Computer was the top subject in the interested category.

Working as Statistician for the IEEE Chicago section, my goals are to work with the Membership Development Chair to meet his immediate needs, as well as explore some ways that we can use the data to help understand our members. In other words, look at our membership at both an aggregate and individual level. In my opinion, some of the best insights will likely come from an investigation of the data at a more granular level, and then determine how findings might apply to the aggregate.

---

Pro Bono: Taproot Foundation - Project Manager (Draft Direction)

Pro Bono: Taproot Foundation - Project Manager (Delivery & Assessment and Closing)

New Book Review: "R in Action"

New book review for R in Action: Data Analysis and Graphics with R, by Robert Kabacoff, Manning Publications, 2011, reposted here:

Quite a few texts on R have entered the marketplace over the last few years, so my decision to go with this text about a year ago was based on my survey of a number of reviews others have written for this and other books in this space, along with my personal experience with other Manning publications such as "Spring in Action", "Java Persistence with Hibernate", and "Hadoop in Action". In general, my experience working through this book has been pleasant, so I personally do not understand some of the complaints that other reviewers have made about the content that Kabacoff provides here.

That said, however, be forewarned that as with other texts, you should not expect to find all of your answers about R in this book. In my opinion, Manning publications are typically written in a format that fits well with the agile learning method with which I have grown accustomed during my consulting career. The author introduces topics along the way, sometimes more piecemeal that I would like, but his style forced me to explore other resources for more detail, bringing familiarity to other available resources. The number of plugin statistical packages for R has grown exponentially over the years (there are now over 2500), for example, so no book, not even "R in a Nutshell: A Desktop Quick Reference (Second Edition)", which I purchased recently, should be expected to be a one-stop shop.

This text quickly brought me up to speed with R language basics working with data sets, and introduced me to specifics with regard to R statistical methods and visualizations. Using R 2.15.0 for Windows, starting with the small data sets the author provides with which to run his examples, as well as sample data sets that the R language itself provides, I soon found myself working with larger data sets that the City of Chicago makes publicly available via its website, followed by using R at work. Your comfort level will be greater or lesser depending on your experience working with data.

As someone new to R, but not new to working with data, I especially appreciated the first five chapters that encompass the first of four parts in the book ("Introduction to R", "Creating a Dataset", "Getting Started with Graphs", "Basic Data Management", and "Advanced Data Management"). Like it or not, but as with any language, most data work revolves around first getting it into the correct format. After these first five chapters, the author walks the reader through basic graphs and statistics, followed by intermediate methods such as regression and analysis of variance (ANOVA), and advanced methods such as generalized linear models and more advanced graphics than was covered earlier in the book.

New Book Review: "Ethics of Big Data"

New book review for   Ethics of Big Data: Balancing Risk and Innovation, by Kord Davis and Doug Patterson, O'Reilly, 2012, reposted here:

A very much needed, extremely well written essay on what the authors call the "forcing function" of Big Data - a push to consider serious ethical issues in light of the usage of Big Data - and how to improve one's organization by utilizing the results of ethical inquiry. In just 64 pages, Davis and Patterson walk the reader through what would have likely been a much more unreachable topic had it been written by another team, and the result is arguably one of the best business texts in the year-2012 marketplace.

After discussing Big Data and Big Data forcing in Chapter 1 ("Big Data, Big Impact"), the authors turn to values, ethical inquiry, and "ethical decision points" in Chapter 2 ("Values and Actions"), followed by a survey of current ethical practices by businesses in Chapter 3 ("Current Practices") and a component-by-component discussion of the the Alignment Methodology Framework introduced earlier in the book in Chapter 4 ("Aligning Values and Actions").

What makes this text so well put together is not just the writing style of the authors, but its organization and inclusion of many examples throughout the discussion that are likely to increase the practicality of the discussion. While Davis and Patterson do formally include a brief case study at the conclusion of Chapter 4, it is the small examples that carry the heavy lifting. And while the Alignment Methodology Framework at a surface level is reminiscent of the familiar Plan-Do-Check-Act cycle, the specifics that the authors provide for each step specifically target organizational values and how to align actions with those values.

Use of the buzzword "innovation" in the subtitle to this effort is actually well placed, unlike the inappropriate use of what can be seen as a buzzword in itself, "Big Data", in some other books in this year's business shelf. As the authors in Chapter 2 comment, "While big data represents both tremendous opportunity (in the form of new products and services) for broad business and social benefit, the opposite side of that coin is that it also represents serious risk. Finding and maintaining a balance between the benefits of innovation and the detriments of risks is, in part, a function of ethical inquiry."

"Developing a capability to find and maintain that balance is partially ethical because of the essential nature of the technology itself...Ethics come into play, in part, when organizations realize that information has value that can be extracted and turned into new products and services. The degree to which ethics play a role in this process is, of course, more complicated that a simple identification of which information is 'ancillary' and which is not. The ethical impact is highly context-dependent. But to ignore that there is an ethical impact is to court an imbalance between the benefits of innovation and the detriments of risk."

New Book Review: "Enterprise Analytics"

New book review for  Enterprise Analytics: Optimize Performance, Process, and Decisions through Big Data, edited by Thomas H. Davenport, FT Press, 2012, reposted here:

This book is a collection of eighteen essays on enterprise analytics by fourteen different authors, ten written or co-written by the editor, who also wrote "Competing on Analytics: The New Science of Winning" with Jeanne G. Harris, and "Analytics at Work: Smarter Decisions, Better Results" with Robert Morison and the aforementioned author (see my reviews), both of whom contributed to this book as well. The topics in this text are a bit more varied than in the earlier efforts of the editor. Rather than focusing on how analytics can be utilized to compete in the marketplace, as was the case in the first book, this latest entry moves in the direction of the second book, which is how to practice analytics. Not in the sense of in-the-trenches toolsets or statistical methods, but at a level more akin to enterprise architecture.

After providing an overview of analytics, the authors direct the reader to topics such as applications and technologies of analytics (approximately 50% of the content), analysts and governance, and case studies. While over a dozen authors contributed to the material, in my opinion it was edited well so that the writing style does not vary to the point of distraction, and the editor reasonably constrained the scope. Chapter 1 ("What Do We Talk About When We Talk About Analytics?") and Chapter 2 ("The Return on Investments in Analytics") were especially well done.

It is about time that someone defined the analytics space, and the table that Davenport provides to break down the three types of "business analytics" (as opposed to other types of analytics, such as "web analytics"), along with the accompanying discussion, is well recommended reading so that everyone can finally get on the same page in this area. There exists some overlap with the table that was provided in Chapter 1 of the second book mentioned above ("What It Means to Put Analytics to Work"), but it is clear that more thought has been put into this subject in the ensuing years. Although the questions that are addressed by analytics have stayed virtually the same, they are more specific in some cases, and the new breakdown into three types of analytics (descriptive, predictive, and prescriptive) from the original two types of analytics (information and insight) in my opinion is much more intuitive.

Along these same lines, I also especially appreciated the definitions and material focused on "engagement" in Chapter 5 ("The Analytics of Online Engagement"), and how one firm measures online engagement with a generalized model it devised. The discussion provided in Chapter 9 ("Analytical Technology and the Business User") on how the multipurpose BI environment of the past, which did not serve business users well, is evolving into environments that are either single-purpose for business users and professional analysts, or multipurpose without the assumption that simplification is needed for business users, was also well done.

The last two words in the subtitle of this book, "Optimize Performance, Process, and Decisions Through Big Data" might be the catalyst for some to pick up a copy of this book, but this is not a book specifically about Big Data. As is noted in the introduction to this text, "This book is based primarily on small-data analytics, but occasionally it refers to Big Data, data scientists, and other issues related to the topic. Certainly many of the ideas from traditional analytics are highly relevant to Big Data analytics as well." My recommendation to readers specifically interested in the Big Data space is to check out the O'Reilly Strata series on Big Data (see my reviews).

Counterpoint: Defining Big Data

About a month ago, I had the opportunity to attend the Big Data in Finance Day Chicago event at Microsoft Technology Center Chicago. One of my many observations was that it is apparent the industry has not agreed on a definition of "Big Data".The best definition that I have personally heard, apart from this event:

• "When the size of the data itself becomes part of the problem"

The speakers at this event gave several:

• "Any data set that is too big or too cumbersome to work with in Excel, or takes too long to calculate.
• "Any data set that has you asking yourself the question, 'Does this belong in Hadoop?"
• "Big Data = transactions + interactions + observations"
• "Big Data is not Hadoop"

The last definition listed above is not really a definition of Big Data, but I list it here because it seeks to define what Big Data is not, and sometimes it helps to provide a narrowing down of options, and it was clear from this event, as it is with so many vendors, that implementations of Hadoop go hand-in-hand with working with Big Data problems.

It is not hard to argue that Big Data is not Hadoop. Hadoop is a solution, not a problem. And besides, Hadoop does not solve all problems in the Big Data space. So what is Big Data?  Let's look at the first definition provided at this event. I have used Excel frequently in the past, but every time I need it for any nontrivial task, I typically need to revisit its limitations.

Excel limitations have decreased over time, making it possible to work with more data. In looking at the limitations for Excel 2003, for example, the size of each worksheet or spreadsheet (a workbook is a collection of spreadsheets) is limited to "65,536 rows by 256 columns". In reviewing the limitations for Excel 2010, however, it is clear that much more data can be stored in each spreadsheet than the earlier release: "1,048,576 rows by 16,384 columns".

But is this really the definition of Big Data? I suppose 16,384 columns is a large number of columns, but 1,048,576 rows is not that large a number of rows, and I have typically seen the need for a far greater number of rows than I have for columns. It is very possible that the speaker summoned the Excel comparison because many business users are familiar with the tool, but I am not sure that this is the right approach.

The source of the "Big Data = transactions + interactions + observations" equation apparently originated with Hortonworks. At the surface, I am not particularly fond of this definition, and the source unfortunately does not explain it well. He gives examples of what he considers transactions, interactions, and observations, but there is some overlap with these areas. For example, it is understandable that interactions might be seen in light of human users, but human users can be the source of transactions as well.

The main problem with this equation is that it does not indicate a threshold in terms of the amount of data that needs to be involved to reach Big Data status. For example, if I have a set of data that involves 2 transactions, 1 interaction, and 3 observations, is the set of data in the realm of Big Data? Perhaps what is not explained is that this equation is not intended to be mathematical, but if this is the case, the definition does not really become any clearer.

Apart from the definition sited at the beginning of this post, which I consider the best definition of Big Data, the defintion that "Any data set that has you asking yourself the question, 'Does this belong in Hadoop?'" is rubbing me the right way as well. This definition points to the size of a data set, which can be assumed to be larger than what alternative tools can handle, so the spirit of this definition is similar with that respect.

In the Review section of "The Wall Street Journal" this past weekend, I happened across an essay by Holly Finn entitled "New Gumshoes Go Deep With Data". Finn mentions an interesting quote by Peter Thiel: "Most of 'Big Data' is a fraud, because it is really 'dumb data'. For the most part, we would need something like artificial intelligence to turn the 'dumb data' into 'smart data', and the reality is that we are still pretty far from developing that sort of artificial intelligence."

In reading the rest of the essay (Peter Thiel is quoted about two-thirds into the essay), I was hoping that this quote would be saved somehow, but alas it was not. Is it possible for data to be smart or dumb? Does data in itself have intelligence? Later in the article, in mentioning Kaggle, the water gets even murkier as Finn mentions the world's best data scientists analyzing "daunting sets of information".

Perhaps this is the source of at least a portion of the confusion surrounding what Big Data is and is not. Data in itself is not smart or dumb. Data does not have intelligence. Data is not information. Data that has been properly cleansed can be the source of valuable information, but it does not provide information in itself. If the vast quantities of data that have become available in the world were information in the raw ready to be plucked to solve the world's problems, there would be no need for the statistician, and there would be no need for the data scientist.

---

Counterpoint: "A doctor's note for virulent consultants"

Counterpoint: "Trojan Horse Consultants"

Google Analytics: 3 Years, 200 posts, 20k pageviews

Just recently, a trifecta of milestones for this blog was reached: the 3-year anniversary of setting up Google Analytics, a posting of my 200th post, and 20k pageviews by visitors. And in looking at the calendar, I realized that it was about time again that I post some observations along the lines of my periodic series focused on Google Analytics. As mentioned in my previous posts on this subject, although Google Analytics was set up for this blog 3 years ago, this blog stretches back another 6 months or so. Visitors from an additional 7 countries have visited this blog since the last update in this series, bringing the number of countries of origin to a total of 124 (refer to the posts I submitted during this time period to see how traffic has grown).

Visitors from the top 10 countries of origin remain the same, with just Netherlands and Spain switching places in the ranking: United States (45.1%), the United Kingdom (4.4%), Canada (4.0%), India (3.8%), Germany (3.3%), Australia (2.8%), Brazil (2.6%), France (2.5%), Netherlands (1.9%), and Spain (1.8%). From another perspective, visits from the top 10 cities show Chicago and London switching places in the ranking, Singapore and Sydney switching places, and San Francisco and Melbourne switching places: (1) New York City, (2) Chicago, (3) London, (4) Moscow, (5) Bangalore, (6) Sydney, (7) Singapore, (8) Melbourne, (9) San Francisco, and (10) Los Angeles.

The top 15 blog pages visited here are as follows: (1) Copycat Advertising Campaign, (2) Anima Sana in Corpore Sano: Part 1, (3) Switching from ASICS to Brooks, (4) Liquibase in the Enterprise: Part 2, (5) Liquibase in the Enterprise: Part 1, (6) Liquibase in the Enterprise: Part 3, (7) Liquibase in the Enterprise: Part 4, (8) Global Leadership Summit 2010 - Day 1: Andy Stanley, (9) RSNA 2010: Quality Improvement? - MRI Workflow, (10) Anima Sana in Corpore Sano: Part 2, (11) New Book Review: "Manage Your Project Portfolio", (12) Treadmill Maintenance, (13) Global Leadership Summit 2010 - Day 1: Dr. Peter Zhao Xiao, (14) "Hello Android": Part 4, and (15) SpringOne 2GX 2010 - Day 2 Keynote. All of these entries were in my last Google Analytics post, with a few blog pages switching places with each other.

While there have been some spikes in the number of pageviews, especially with some of my recent posts on Big Data related topics, the stability that I outlined in my 2-year anniversary blog post remains much the same, and the share of visitors from the United States expectedly continues its downward trend.

The in-page analytics report reveals that 12.8% of visitor clicks were either on the title of my blog at the top of the page (which brings users back to the landing page) or the next page link at the bottom of the page, at least 6.8% of visitor clicks were in the links to the archives on the left side of the page, and that at least 11.2% of visitor clicks were in the cloud categories area on the right side of the page. These three areas apparently comprise at least 30.8% of clicks once visitors land on my blog. In order to calculate the exact figure, I would need to dive deeper in the data, since some of the bubbles in this report overlap with one another.

Big Data in Finance Day Chicago (September 18, 2012)

Personal notes from Big Data in Finance Day Chicago that I recently attended at Microsoft Technology Center Chicago. In speaking with event organizer and Fountainhead owner Andrew "Shep" Sheppard (a self-described quant by day and meetup organizer by night), following the last session, he mentioned that the first Big Data in Finance Day was held in New York City on March 26, 2012, and elaborated on his plans by indicating that the next conferences in this series will be held in Boston and London.

After finding out about this conference through Chicago Big Data, I offered to do a write-up about the conference on my blog, similar in spirit to the one I had posted following my attendance at BigDataCamp Chicago, in exchange for fees being waived, so I attended all 14 sessions of this marathon day-long event. By the time mid-afternoon came around, a number of attendees had already left, but I am glad that I held out, as one of the best sessions of the day, led by Dr. Phil Shelley, CTO of Sears Holdings and CEO of MetaScale, occurred late in the day.

The alternate name for this event was Hadoop in Finance Day Chicago. It will not take long for anyone in the Big Data space to realize that a summoning of the open source Apache Hadoop product is often performed alongside mentions of Big Data because of its ubiquitous influence in projects of this genre, as well as its ubiquitous presence in commercial products in this space. While most of the sessions included Hadoop discussions, working with Big Data goes beyond what any one product can solve, and so the number of topics was broad.

---

8:45am - Registration
8:45am - Welcome address
9:00am - "Big Data in Finance by the Numbers: Data about Big Data"
9:30am - "Big Data at CME Group: Challenges and Opportunities"
10:00am - Break
10:30am - "Big Data in Finance: A Regulator's Perspective"
11:00am - "Hadoop: A (Very) Brief Introduction"
11:30am - "Hadoop in Financial Services"
12:00pm - Lunch
12:00pm - Lunch and learns: "MATLAB and Hadoop" and "An Integrator's Perspective: Big Data in Finance"
1:15pm - "Big Data Meets Cloud: The Perfect Arranged Marriage"
1:45pm - Panel discussion
2:30pm - "OK, So You've Decided to Put Data in Hadoop. Now what?"
3:00pm - Break
3:30pm - "Scalability in Hadoop and Similar Systems"
4:00pm - "Scaling Out Excel Using Windows Azure and Hadoop"
4:30pm - "Data Analysis with Hadoop"
5:00pm - "How VoltDB is Making Hadoop a More Productive Tool for the Enterprise"
5:30pm - "Raw Data to Real-time Decisions: Feedback Loops Using CEP and Hadoop"
6:00pm - Wrap-up
6:15pm - Drinks reception

---

---

"Big Data in Finance by the Numbers: Data about Big Data"

"Data in finance is growing quickly, driven by a number of factors from increased trading volume to new and far reaching regulatory mandates. This talk will discuss (and quantify) where we are with big data in finance, and more importantly, where we seem to be heading."

Agenda:

- golden age of computational discovery
- what is Big Data anyway?
- let's look at the numbers
- what's driving this?
- where do we go from here?
- a cautionary warning (Big Data: a health warning!)

- Andrew Sheppard, Fountainhead

Shep started his talk by presenting a draft sketch that he had created for O'Reilly a while back of computational power available to the individual throughout history. The graph showed an expansion of CPU and CPU mulitcore, GPU, CPU Cloud, and Cloud + GPU during the last 10 years (2000-2010). The speaker declared that it is a golden age of computational discovery across the board, including in finance.

What is Big Data? "Any data set that is too big or too cumbersome to work with in Excel, or takes too long to calculate." With Microsoft Excel 64-bit, this means data in the 10's of GB range. What is Big Data? "Any data set that has you asking yourself the question, 'Does this belong in Hadoop?" There are 3 dimensions of data: volume (or quantity), complexity (or variety), and speed (or velocity).

According to IBM, 90% of all the data in the world was created in the last 2 years. How many years until Big Data spending dominates IT spending? According to one estimate, 8 years. The size of the market for financial data is $23.2B, probably the most organized data market. Counterparty Valuation Adjustment (CVA) is a Big Data and big compute problem. Typical bank-wide CVA may involve millions of trades; some complex derivatives that are expensive to price. The half-life of financial data can be quite short.

Shep quoted someone who said that most of the time they spend drinking, and the rest is a waste of their time. He used this as an analogy to say that soon we should be doing Big Data most of the time, because otherwise the rest will be a waste.

There are 4 factors driving Big Data growth: technical capability (computing cost), greed (opportunity), feat (competitive advantage), and regulation. The number one factor is making money, and the second is fear. One thing that is often overlooked is the fact that analysis itself is a source of Big Data. In some cases, derived data can dwarf the original data set.

Examples in finance include High-frequency Trading (HFT), news bots, scenario analysis and stress tests, and data sets that often feed on themselves. Certain trends have been established and will continue: trading volumes will continue their rapid growth, and machine trading (trade that involves machines trading autonomously) will become the norm.

Data standards and formats will improve. And there are social implications with Big Data. Big Data in many ways is being redefined as Hadoop in the sense that these two are more often than not in the same sentence. Hadoop is just one NoSQL data store. We are going to see more domain specific languages (DSLs) for Big Data.

Enormous economic value is and will be extracted from Big Data, otherwise Big Data would not be growing. There are regulatory factors. Better data life cycle models, from creation to deletion are needed, along with data cleansing and scrubbing. In short, there are huge challenges, but also huge opportunities. The Big Data market is expected to grow to $16.9B in 2015.

Apophenia: "The experience of seeing meaningful patterns or connections in random or meaningless data. The term was coined in 1958 by Klaus Conrad, who defined it as the 'unmotivated seeing of connections' accompanied by a 'specific experience of an abnormal meaningfulness'."

Remember this: "Without data, it is just an opinion!" Shep noted that the Hadoop vendors are "attending today in full force" because work is not being done in a vacuum to develop solutions. Creating actionable knowledge is still in the baby phase.

One attendee asked what has really changed, since finance has been working with significant amounts of data for quite some time. Shep responded by saying that what has changed is the sheer velocity and speed of data. Hadoop has become important because the amount of data has increased so rapidly. Shep argued that data is more or less random, and there are tests to determine the extend of randomness, but this is a topic that could consume an entire day in itself.

---

"Big Data at CME Group: Challenges and Opportunities"

"Find out how one company is meeting the challenges and opportunities of Big Data."

Agenda:

- CME group overview
- Big Data at cme group
- Big Data use cases at CME Group
- Big Data challenges and opportunities at CME Group
- Big Data key learnings at CME Group

- Rick Fath and Slim Baltagi, CME Group

Rick did most of the talking during this session, and Slim joined Rick and others for the discussion panel later in the event. He started by reviewing at a high level where CME Group is at today, including the fact that it is the number one exchange in the United States, and globally by 2011 trading volume with 11.8 million contracts per day. The company has a strong record of growth, both organically and through acquisitions.

The speaker mentioned that he was "one of the fortunate ones" to attend the Hadoop Summit this year. While Shep offered two different answers to the question "What is Big Data?", Rick offered a third: "Big Data = transactions + interactions + observations". A massive amount of transactional data important for business.

His group had to learn Hadoop on their own, because not a lot of books or training was available. This comment is similar to a comment that Shep made earlier, that a lot of the time problems he works on involve stuff being thrown at him that he is asked to just figure out. As a consultant, I can relate.

Data is not merely a byproduct at CME Group, but is an essential deliverable. Daily market data generated by the exchange, as well as historical data. Growing partnerships around the world. The message here was that the volume of data is always growing, so leveraging legacy tools versus adopting new Big Data technologies needed to be investigated. Not all data was being stored, and quality was lacking, and they needed to learn from this lesson. Rick offered three use cases.

Use Case #1: CME Group's BI solution. Provides complex reporting and data analysis for internal business teams. Challenges: high performance is needed (existing queries were taking 18+ hours to execute), as well as working with established tools and partnerships. (The speaker went through this use case quickly, so I did not have enough time to get down more notes.)

Opportunities: Structured RDBMS, minimize integration impact, enhance customer analytics, deliver faster time to market, improve and increase fact-based decision making, and reduce execution duration of batch processes. Why they chose a data warehouse appliance: no need to reengineer Oracle database. (The speaker went through this use case quickly, so I did not have enough time to get down more notes.)

Use Case #2: CME Group's rapid data platform. Reduce reliance on SAN while increasing query performance. Challenges: expensive storage, Oracle queries cannot keep up with inserts and do not meet SLAs. Opportunities: reduce storage cost. (The speaker went through this use case quickly, so I did not have enough time to get down more notes.)

Use Case #3: CME Group's historical data platform. Provide historical market data to exchange customer challenges. Challenges: expensive storage cost, legacy downstream application with limited support, historical data from acquisitions and mergers ("don't look for problems, because you will find them"), 100TB and growing, data redundancy and quality (limited awareness).

Opportunities: Reduce storage cost with non-specialized hardware ("not commodity"), reduce duplication of data, improve data quality with better interrogation, grow the business (new datasets, mashups, analytics), separate delivery from storage ("store everything and deliver what you need"), enable business users (ad hoc queries, define new datasets), reduce TCO (support, improve reliability).

Why was Hadoop chosen? Solution built for scale and growth, structure of data does not matter ("data is the data"), Hadoop as ETL solution, performance POC with Oracle (the reason the project got funded was because it beat Oracle queries, and removes duplication), decouple storage and delivery ("store raw data, deliver enhanced data"), ecosystem reduces development time and provides solutions to common problems, structure is fluid, perfect for historical data.

But there are enterprise challenges. Open source paradigm (infrastructure counterparts start to be brought into the room), enterprise maturity built for failure ("don't mention 'built for failure' to the infrastructure guys"), operations readiness, framework can mask problems, educate the enterprise about Hadoop, evolution of Hadoop.

Deployment challenges include selection of Hadoop distribution, growing cluster equals growing support fees (how to lower maintenance fees), future project candidates (super cluster versus individual cluster), distributions support and alignment, maximize investment in its assets. (The speaker went through this material quickly, so I did not have enough time to get down more notes.)

CME Group learned to embrace open source, enterprise readiness, solve operational maturity, leverage existing IT investments, adapt standards to make cluster more supportable, tackling the learning curve (keep close to the community), leverage the Hadoop ecosystem, capture everything for integrity purposes etc by keeping it raw (data does not need to be structured), capture data first and then figure out opportunities (opportunities might not be there now, but may be in the future, Hadoop can be complimentary to existing IT assets).

Apache HBase was chosen for the first use case because it involved a historical data system not requiring speed. The other use cases required high performance, so used HDFS. In industry, 80% of Hadoop users use HDFS, 20% use HBase and Apache Hive. There is no one size fits all, and Hadoop may not even be the right solution for some applications. "Big Data is not Hadoop."

Most of the data at CME Group is structured, but the affect of the news on trading patterns etc requires analysis of unstructured data. CME Group production cluster is 15 data nodes, about 50TB, with up to 384GB of RAM per machine, so not using commodity hardware in this case. For marketing purposes, Hadoop is supposed to be best on commodity hardware, but this is not always the case. HBase is much more stable now, much less risk than a year ago, so things are moving fast in terms of maturity.

---

"Big Data in Finance: A Regulator's Perspective"

"Big Data more and more falls under the purview of the regulators. You'd better understand the issues before embarking on that Big Data project."

Agenda:

- preamble: "a regulators lot"
- what is regulation anyway?
- how is regulation driving big data?
- an example: bank-wide cva
- who regulates what? an american perspective
- a look at dodd-frank
- a parting question

- Andrew Sheppard, Fountainhead

Shep noted that he tried to get a regulator to speak at this event to no avail. He was able to get one to speak for the event held earlier in New York City, but that individual has since decided to move on as a professor at MIT.

The speaker started the discussion by exclaiming that "this is the hardest talk of the day". Regulators have to be reactive rather than proactive: "first do no harm". If you thought your "corporate relations" department was difficult to deal with, try the government! Winston Churchill was quoted as saying that "democracy is the worst form of government...except all the others that have been tried".

From the perspective of a regulator, Big Data in finance is both a boon and a liability. Boon because it offers the possibility of better and more timely reporting of the numbers with an audit trail. Liability because events of importance may be lost, or indeed hidden, in vast amounts of data. When things go wrong, it is often only the data that tells a complete and impartial story of the events.

Working definition of "regulation": "the purpose of financial regulation is to supervise and oversee financial markets so that they function properly and efficiently, and specifically, to maintain integrity of the system so that major dysfunction and crises are avoided."

Shep mentioned that there are three regulators in the audience today. Regulations, by their very nature, are mandates: regulators want more timely reporting of data, regulators want better, more accurate, numbers, regulators want an audit trail, regulators want accountability.

Example: bank-wide CVA. CVA is both a Big Data problem and a big compute problem: 10m trades, 10k Monte Carlo paths, 100 time steps - that's 10^3 valuations to get just one CVA number! Will likely be mandated by the regulators in the next couple years. Basel III and the cost of capital.

Many countries have a single regulator. The United States has many: SEC, FINRA, CFTC, the "Fed", FDIC, OCC, NCUA, OTS. Shep offered the reminder that one needs to adhere to mandates if one is doing a Big Data project in finance.

The Dodd-Frank Act is 1k pages of new regulations: Title VII of the Dodd-Frank Act is a technological overhaul of the derivatives markets. Dodd-Frank effectively overhauls trading technology from the 19th to 21st century, risk mitigation technology from decentralization to Central Counterparty Clearning (CCP), data and documents (human-readable to machine-readable), and transparency to regulators, to counterparties, to the market.

New data required for Dodd-Frank includes clear data, messages, trade capture, market feed, reports to markets, regulators, and the public. If data is not real-time here, its worth is minimal

Pre-Dodd-Frank, it is not like no data was being captured, but different data is being captured, and on a more timely basis. All of the old stuff, plus the new stuff. Post-Dodd-Frank requirements include more machine readable, more standardized, and more easily available, more trade data, more forms/reports, more formats/representations/dictionaries, more connections to data sources/access issues, more quality control issues, more data security issues, more legal issues.

How can a Big Data approach help? It helps with data capacity/storage/retrieval, data access/security, data processing, data analysis. It is not all bad news.

Shep asked attendees, by show of hands, if we had more data and more compute power going into the recent financial crisis, whether the outcome would have been less severe, the same, or much worse. Of an audience of hundreds, 2 attendees voted that it would have been less severe, about one-third of the room said it would have been about the same, and about 10% to 15% of the attendees said it would have been much worse.

The speaker noted that when he had asked this question at the New York City session, not one hand was raised for "less severe". The question that he then posed was that if nobody seems to believe that the outcome would have been "less severe", why are we investing in all of this technology?

Do the regulators have the skills to understand the data being handed to them? Shep mentioned that someone from FINRA mentioned before his talk today that we are the best ones to understand the data - the folks that actually created it etc.

Shep mentioned that he really does not know how much it will cost to address Dodd-Frank, but he expects it to be very expensive. As someone working in the trenches, he thinks that we will be feeling our way. Even with an estimate, there will be a huge spread depending on company focus.

---

"Hadoop: A (Very) Brief Introduction"

"Hadoop is more than a single tool, it is the hub in a wheel of complementary tools (Hive, Pig, HBase, Mahout, and others). This talk gives an overview of the Hadoop tools set."

Agenda:

- high-level overview of hadoop
- deeper dive into hadoop components
- the hadoop ecosystem

- Jonathan Seidmann, Solutions Architect/Partner Engineering at Cloudera

Note that I took very few notes during this session, because I was already familiar with the bulk of the information presented.

Apache Hadoop 2.0 now supports high availability. Data nodes use local storage in a JBOD (Just a Bunch of Disks) configuration, not RAID, which allows Hadoop to round-robin I/O requests. New Hadoop releases address NameNode crashes, a problem with earlier releases.

The Hadoop ecosystem according to Jonathan includes more tools than often presented, and includes Apache Hive, Apache HBase, Apache Pig, Apache Mahout, Rhadoop, RHIPE, Cloudera Hue, Karmasphere Studio, Cloudera Manager, Apache (Incubator) Ambari, Apache Sqoop.

Currently, Hadoop provides manual failover to the standby NameNode, but soon an automatic failover will be made available.

---

"Hadoop in Financial Services"

"Hadoop is already finding wide application in financial services, a trend that looks set to expand considerably in the coming years. This talk covers how Hadoop is currently being used with financial Big Data, and where it might find even wider application in the future. Basically, how financial Big Data fits together with Hadoop."

- Adam Gugliciello, Solutions Engineer at Datameer

Datameer is a business intelligence (BI) tool that runs in Hadoop. Most use cases that Adam sees are not possible with the old paradigm: ETL, data warehouse, BI. If any of these pieces change, you need to go back to the drawing board to reevaluate. What the speaker has typically seen is that 6 to 12 weeks is typically required to act on a change in these areas.

Hadoop and Datameer enable fast raw loads, the ability for data scientists, analysts, etc to define the problem to be solved, etc, and self service including drag and drop, spreadsheets. These common themes ran throughout all of the use cases he presented during this session.

Use Case #1: Internal asset movement. Focus: Institutional risk. Identify departmental asset movement. Actual "rogue trader" identified using Datameer. Craft an early warning system for future detection. Aggregate sources and correlate. Net result: Reduced risk.

Use Case #2: Client risk profiling. Focus: Bank assets. Identify profile-scoring features in loans and mortgages. Build qualifying models with Datameer. Test models against existing data. Provide recurring reports. Focus: Client assets. Monitor risk across product offerings for all client assets. Develop common base of client characteristics for models. Analyze risk types: default, liquidity, attrition risk. Monitor: Transactions, web, email, chats, and call logs (all of the different ways that clients interact with the bank). Net result: Correlation and increases agility.

Use Case #3: SLA analytics. Focus: Infrastructure performance. Companies are using Datameer to aggregate all logs to Hadoop. Transform log data into actionable insight into IT systems performance. Measure conformance to existing SLAs. Reduce the batch window. Net result: Reduced costs.

Use Case #4: Security fraud. Focus: Client protection. Run statistical models against customer profiles. Detect anomalies with pattern recognition algorithms. Identify and investigate identity theft. Detect and prevent fraud. Protect customer assets. Net result: Protected customers.

Use Case #5: Data quality and profiling. Focus: Profiling. Use profiling techniques to assess and score client accounts. Run complex data quality rules and models. Build exception reporting. Focus: Data quality. Enable data stewards to run ad hoc analysis and dashboards. Drive iterative processes to refine data. Net result: Reduce risk and cost.

Use Case #6: Security infrastructure. Focus: Infrastructure protection. Monitor external intrusions. Detect suspicious traffic. Net result: Increased security.

Use Case #7: Dark pools and cost savings. Focus: Trade and partner analysis. Financial institutions often hold large positions that they want to trade in ways that do not affect public markets. Dark pools are private venues that enable trading. (The speaker went through this material quickly, so I did not have enough time to get down more notes.) One caveat here is that the speaker mentioned that he has not spent a lot of time in this area.

Adam commented that Datameer and Hadoop bring commoditization, and data out of the back room. Mainframes and RDBMS are not going away. While RDBMS disrupted mainframes, Hadoop disrupted RDBMS. Datameer is used to prepare data for Cognos etc, and is also used for the data analysis layer etc. Datameer uses array analysis rather than set theory analysis.

---

"MATLAB and Hadoop"

"A first look at methods of bringing MATLAB numerical analysis to large data through Hadoop. During lunch, Eugene McGoldrick, Director of Development for Financial and Application Development from MathWorks, Inc. will discuss options for integration of MATLAB execution on Hadoop infrastructure."

- Eugene McGoldrick 

MathWorks is turning MATLAB into a financial development environment. MATLAB provides numerical analysis, data analysis and visualization, algorithm development, and application development.

Mathworks provides financial products that run on top of MATLAB to do financial analysis etc. MATLAB compiler and builder products automatically packages your MATLAB programs as standalone applications, software components, and targeted programming/application environments. The full MATLAB language is supported allong with most toolboxes. Allows for royalty-free deployment. You can run Java etc components outside of MATLAB environment once compiled.

MATLAB Hadoop integration. Hadoop invoking compiled MATLAB Java components and compiled MATLAB production server components. MATLAB invoking Hadoop tasks/jobs. MATLAB Production Server is a new product in 2012. Directly deploy MATLAB programs into production. Scalable and reliable. Use with web, database, and application servers.

Eugene explained that it takes about 5 lines of code to integrate into Java, and therefore Hadoop. The Parallel Computing Toolbox is not needed to do this, so there is some cannibalization with this product. MathWorks is at the early stages of MATLAB and Hadoop integration, so there is still work to do.

In response to a question from the audience that commented it must have taken a long time to get this functionality built, the speaker indicated that it took him only about 2 months from concept to implementation to get this to work. He also siad that you do not need to do any JNI coding. Compilation needs to be performed against each platform. The latest version of MATLAB runs on Java 7.

---

"An Integrator's Perspective: Big Data in Finance"

"Integrators of Big Data have the advantage of implementing projects across different areas of finance, and across different functional departments in large institutions. Often the challenges are more than just technical. Hear what one integrator has to say about the challenges and the opportunities and how to successfully integrate Hadoop and other technologies into your Big Data projects."

Agenda:

- market
- services
- cases

- Jim Buonocore, Alliance Global Services

The speaker started by stating that one of the biggest hurdles on integration projects is people. Regarding Big Data usage, the top spot goes to digital content, followed by Web 2.0, retail (bricks and clicks), and legacy transactional (which includes financial services). I am not sure how the speaker defines "Web 2.0", since the term has been overused, and is a bit old.

A lot of Web 2.0 companies decide to start with Hadoop at the outset. In the legacy transactional space, Hadoop is used in pockets. Infrastructure characteristics include outlier, isolated, largely uncoupled, highly purposed, substantial replication, a priori controlled, proximate outcomes. Regading digital content and Web 2.0, Hadoop is used in platforms. Infrastructure characteristics include predominant, open, ubiquitously coupled, amorphous, organically integrated, lower control levels, proximate outcomes.

The speaker noted that much of the info he is sharing is as much fact as it is personal opinion. At Alliance Global Services, the sales team struggles to sell in some aspects, because it is not another CRM etc - it is unfamiliar territory to many.

The largest spend category is integration, followed by infrastructure and COTS products. Integration involves infrastructure connectivity, product configuration, data services, application connectivity, portfolio event management, extra-enterprise connectivity, intra-platform product placement, core build, operations.

Service providers and consulting firms typically needed. For service categories, build capacity takes the top spot, followed by tool expertise, domain expertise, operate/maintain, infrastructure, control/risk, PM/PMO, strategy, and choosing vendors.

Regarding technology directions, the current buzz is all about the platform, the technology, and differentiators. In the near term, significant growth is seen in open platforms, rapid maturation, hybrid technology stacks, heterogeneous environments, lagging growth in mature industries. Long term technology directions includes functional emphasis (platform abstraction), tool independence, commodity platform availability.

Wholesale adoption in legacy industries. Scope for all types of Big Data computing. Easier for legacy C-levels to buy function. Ubiquity. Fully integrated, tool independence, hybrid technology stacks, heterogeneous environments. The speaker went over three use cases on Master Data Management (MDM/Reference), operational performance and risk, and "post doc" research support.

---

"Big Data Meets Cloud: The Perfect Arranged Marriage"

"Big Data, many say, belongs in the Cloud. Hadoop will be there to receive it! This talk discusses the advantages of putting financial data in Hadoop in the Cloud."

- Abhishek Mehta, tresata

According to Abhishek, tresata is the first vertically focused Hadoop analytics company. It uses Big Data as a service business model, gaining actionable insights driven by client/market behavior.

The speaker commented that "cloud" should not be mentioned to a bank for good reasons. Take aspects of the cloud, and deliver solutions. Clients are sold by features of the cloud, such as pay-as-you-go etc, but not by the term "cloud" itself.

Acknowledge the risks. Any data privacy issues can kill you. Clients such as banks want to keep their data separate. Interoperability bringing more accurate predictions does not apply, because interoperability is not an option. "You can watch, you can touch, but can you keep?"

In the B2B space, client data is not owned by tresata, but if data is combined from multiple sources to create something new, it does own it. The speaker tells clients that data is always owned by them, but the software is not.

The concept of the cloud must handle situations where data is constrained to live in certain countries, such as Luxembourg, where data cannot leave the country. The speaker commented that most of the issues he discussed during this session go away if data is anonymized. If data sits within own firewall, anonymization is not a requirement. Consumer data has been sitting in the cloud for years, such as consumer data used to calculate FICO scores.

Abhishek posed the question: What is the purpose of you moving into the cloud? He commented that for analytics, one needs to be careful. You need to ask questions around privacy and ownership. On the B2B side, the rules are fairly clear, but on the B2C side, these are not.

There is a demographic aspect that cannot be controlled. There is also an educational aspect, which is lacking with regard to "free" services. With "free" you always have to give something up. The race that is going on is how much data you can bring to the problem, but the problem is how you do it.

---

Panel Discussion

"Viewpoints, opinions, and Q&A with a panel of experts."

Tom Groenfeldt, moderator
Slim Baltagi, CME Group
Adam Gugliciello, Datameer
Andrew Paul Acosta, Milesius Capital Resources
Abhishek Mehta, tresata
Krishna Nimmagadda, MetaScale
Tedd Dunning, MapR Technologies
Wen-Ming Ye, Microsoft
Ari Zilka, Hortonworks
Carmen Taglienti, BlueMetal Architects

The moderator started this discussion by asking: What is it with Big Data that is so new that we couldn't do before? What is my ROI? A lot of banks have 40 or 50 systems that don't even know what they have, so the first step is to make this determination. Do you see anything that will change the world?

Ari Zilka, Hortonworks: Every single vertical can now do data retention, we're not leaving data on the floor, and we're doing full table scans.

Adam Gugliciello, Datameer: I'm not seeing anything yet, but the death of the star schema, the ERD, etc are dying. If you're not storing normalized data, you can look at the original data.

Slim Baltagi, CME Group: There is now the opportunity of doing more with data, away from traditional data silos, and the opportunities to do realtime work are increasing.

Andrew Paul Acosta, Chief Data Scientist, Milesius Capital Resources: Now it's about business value. We ask executives what they would like to do if they were not constrained by master data models etc.

Wen-Ming Ye, Microsoft: Human creativity is driving innovation, beyond what technology is able to do for us.

Carmen Taglienti, BlueMetal Architects: Moving forward, more firms will be using more data, especially in regard to sampling, where all the data can be used, not just samples.

Krishna Nimmagadda, MetaScale: Clearer ROI models will help enable funding when you're offering predictions that will drive revenue etc.

Abhishek Mehta, tresata: Every model (predictive etc) that has been written will be rewritten.

Tedd Dunning, MapR Technologies: Reducing costs is not going to change anything, but growth patterns will.

How do you go about selling and explaining Big Data?

Slim Baltagi, CME Group: Took approach of almost begging from department to department, and got them more involved. Manager did a good job at going to upper management. It is human nature to be resistant to change.

Ari Zilka, Hortonworks: I don't sell Big Data or Hadoop. The line of business wants to answer certain questions, and Hadoop helps answer those questions. In general, the goal is not to oversell. I don't educate on Big Data, they educate me.

Abhishek Mehta, tresata: A massive driver is the fear of being left behind. Greed is another driver, as is economics.

Krishna Nimmagadda, MetaScale: (Didn't really answer the question.)

Tedd Dunning, MapR Technologies: (Went off on a tangent to respond to Abhishek Mehta, who said that social media cannot be predictive.)

Abhishek Mehta, tresata: Responded to Tedd by saying that he didn't say social media cannot be predictive, but that the models have not been written yet.

Tedd Dunning, MapR Technologies: Explained why Abhishek Mehta is wrong, and gave examples of models that have already been written.

Wenming Ye, Microsoft: Demos or POCs can be used to demonstrate the value of Big Data. At Microsoft, one problem is that they already have a lot of in-house solutions, and the question is posed as to why they should invest any more money in this area.

---

"OK, So You've Decided to Put Data in Hadoop. Now what?"

"This talk covers some of the issues you will face as you move data to Hadoop and manage it there. Also, you need to integrate Hadoop with other data (because not everything belongs in Hadoop), and this talk discusses the challenges and solutions of integrating with legacy data."

- Dr. Phil Shelley, Sears Holdings, CTO, and MetaScale, CEO

Dr. Phil Shelley, CTO of Sears Holdings and CEO of MetaScale, started his presentation by indicating that Sears Holdings has about 4k stores in the United States. They wanted to do things with data that they could not do, such as loyalty programs etc, pricing algorithms based on inventory etc. Reports were being generated with Teradata on a quarterly basis that took 6 weeks to run. The business wanted to start running these reports weekly.

They were having issues with meeting production schedules. Multiple copies of data, no single point of truth. ETL complexity, cost of software and cost to manage. Time to take to setup ETL data sources for projects. Latency in data, up to 2 weeks in some cases. Enterprise data warehouse unable to handle load. Mainframe workload over consuming capacity. IT budgets not growing, but data volumes escalating.

Phil commented that a solution always involves an ecosystem, so you are not going to solve all your problems on one platform. Sears Holdings has a huge sprawl of products in this space because you cannot put them all in one place.

Keeps about 90 days of data (consisting of box scan data etc) within Teradata. Regulators need 10 years, but it is cost prohibitive to keep all of this data in Teradata. There are speed, storage, and cost factors. It does not make sense to store everything in Teradata when you are only going to query it once in a while. With Hadoop, it is now economically feasible to do this. The single point of truth is now a reality with Hadoop, only moving data when you need to do so.

Sears Holdings has an ESB for point of sale transactions etc. Half of the workload still originates in mainframes. Short 5-minute batch Hadoop jobs are run throughout the day from credit card transaction systems. What they have focused on is a single point of truth for analytics. The speaker pointed out that they do not allow any more new ETL.

About 10 hours of ETL have been reduced to 17 minutes with Hadoop, and if there is a failure, they just rerun. About 5k lines of COBOL have been replaced with 300 lines of Pig. Much more maintainable. Some IT projects have paid for themselves within a year. Some even in a month. Phil commented that he has never seen results like this before. In some cases, the cost of 1 MIPS is $10k per year, so for a system with 10k MIPS, he commented that it is not that hard to see the cost savings when transitioning off of a mainframe.

Rethink everything. The way you capture data, the way you store data, the structure of your data, the way you process data, the way you analyze data, the costs of data storage, the size of your data, what you can analyze, the speed of analysis, the skills of your team. The speaker mentioned that he has some folks with 30 years of mainframe experience that worship the mainframe, and this has been an eye-opener for them. Companies need to experience these capabilities themselves.

A number of lessons learned at Sears Holdings. Big data tools are here and ready for the enterprise. You can move mainframe workload to Hadoop and run jobs up to 50 times faster! You can mostly eliminate ETL with Hadoop. An enterprise data architecture is essential. Hadoop can handle enterprise workload, reduce strain on legacy platforms, reduce cost, and bring new business opportunities. The solution must be an ecosystem, part of an overall data strategy, and must not be underestimated.

Cubing is no longer needed. Data is just read and aggregated when it is needed. It took his team about 2 years to move to Hadoop. They used to use SAS on the mainframe, and now use R and Hadoop. The folks using Hadoop etc at Sears Holdings were retrained, and all these folks came from in-house. Brighter data architects and developers took about a year to get up to speed.

The reason they went with Datameer for part of their ecosystem is because it is a good way for people to explore data, without the limitations of Excel etc where you cannot import 100TB of data when running a query.

Slim Baltagi from CME Group chimed in from the audience and said that he disagrees that these tools are 100% enterprise ready, and the speaker said that they make it work. Hadoop is the first tool that provides a real data hub where data can be stored and analyzed in the same place. A type of ETL is used, but without the E and the L.

---

"Scalability in Hadoop and Similar Systems"

"Wherever you store and analyze your financial Big Data, scalability and performance is key. Hadoop scales and performs well out-of-the-box, but there is much you can do to make the most of your hardware and improve both scalability and performance."

- Ted Dunning, MapR Technologies

Note that Ted gave a great presentation that was essentially the same as the one he gave at BigDataCamp Chicago, so I took very few notes during this session. In my opinion, this presentation is very compelling - one of the best at both events.

Why Big Data now? If it were just availability of data, then existing big companies would adopt big data technology first, but they did not. If it were just a net positive value, then finance companies should adopt first because they have high opportunity per byte, but they did not. Under almost any threshold argument, startups would not adopt Big Data technology first, but they did.

Someone from the audience commented that Hadoop does not solve all problems. Ted responded by saying that the fact that the facts that Hadoop does not like to connect and does not like mashups can be remedied, and the fact that Hadoop does not like realtime can be remedied. Someone else suggested that Hadoop will disrupt free markets, but Ted responded by saying that this is already being done. Anyone who can disrupt free markets is the new status quo.

Ted furthered in his responses to audience comments by saying people have observed that the law of large numbers does not always work. This has been observed many times, but people keep going back to normality and ignore the theories. We are definitely communicating at a level we have never been able to do in the past.

---

"Scaling Out Excel Using Windows Azure and Hadoop"

"Excel is the de facto front-end choice in finance. From the trading desk to the risk department, Excel is ubiquitous, and for good reason: it is a data analysis tool that everyone knows how to use. This talk shows how to use Excel as a front-end to Big Data in Hadoop for financial analysis."

Agenda:

- Excel and Hadoop on Windows Azure
- Excel 2013 with PowerPivot and PowerView
- Excel compute with Windows HPCS (HPC Server), Windows Azure

- Wen-Ming Ye, Senior Technical Evangelist at Microsoft

Wen-Ming started the presentation by commenting that you can try out online what is probably the only currently available free Hadoop cluster: www.hadooponazure.com

The speaker went over at a very high level the following, all centered around Microsoft technologies: HPCS (burst/cloud numerics runs on top of HPCS), the Microsoft distribution of Hadoop, data mining/predictive analytics, data marketplaces, PowerPivot (Excel self-service analytics), PowerView (Excel 2013 data visualization), and business intelligence: analytic services and reporting services (SQL Server).

An ODBC driver is available for Hive, as well as a Hive add-on for Excel and integration with PowerPoint. A Hadoop based distribution for Windows Server and Azure is available through a strategic partnership with Hortonworks. A JavaScript framework for Hadoop is also available from Microsoft.

The speaker gave demonstrations of www.hadooponazure.com, using Excel to access Hadoop with Hive, Excel 2013, and HPCS. Wen-Ming commented that for Big Data, make sure you get the 64-bit version, and that PowerView was really made for nontechnical business users. (The speaker went through this material quickly, so I did not have enough time to get down more notes.)

---

"Data Analysis with Hadoop"

"Analysts are increasingly interested in new types of unstructured data – sentiments expressed online, and text from newswires and social networks, and others. Exploiting these new data sources is leading to new insights and new business opportunities. Hadoop forms a data 'refinery' that when integrated with traditional data warehouses and using Hadoop's ecosystem of analytical tools unlocks patterns that underly the data."

- Ari Zilka, Chief Products Officer at Hortonworks

Ari mentioned that he was also the founder of Terracotta, a low latency Big Data platform, about which I saw him speak at SpringOne 2GX 2010.

Integration with traditional data warehouses is what Ari mentioned was the key for this session of the event. Within 30 to 45 days, we should hear about the Microsoft and Hortonworks collaboration, but he indicated that he would not discuss during this event. As Rick Fath from CME Group indicated in an earlier session, "Big Data = transactions + interactions + observations". Companies that can connect data between storage silos can unlock a lot of insight.

The speaker briefly discussed three patterns: Refine, Explore, and Enrich. The Refine pattern looks at the data and sees something new that had not been seen before. With the Explore pattern, questions start being asked, and with the Enrich pattern comes better predictions.

The Refine pattern is an operational data refinery, and data needs to be cleaned as soon as possible. Ari gave an example of a "Big Bank" service with two users and a pipeline, with 10k files being dumped every day to GPFS, the IBM clustered file system. According to the speaker, using GPFS as a refinery was not so good, because it only contained 1 day worth of data that was initially imported (only 1/20 of the columns) directly to Teradata. HDFS was brought in to store 3 to 5 years of data rather than 1 day, and connected to Vertica, Aster, Palantir, Teradata etc that detected and uploaded only changed records in HDFS.

With the Explore pattern, Hadoop is used as an agile, ad hoc data mart. An example was given of a hardware manufacturer unlocking customer survey data (think SurveyMonkey). Survey data was joined with CRM data etc. The Enrich pattern delivers results of a Hadoop analysis to online apps. An example was given of the visits to custom web pages of a clothing retailer. Hadoop was used to perform cluster analysis of this data stored in HBase.

The speaker also mentioned some Aster and Hadoop patterns in analytics that covered path analysis, text analysis, statistical analysis, segmentation, marketing analytics, and data transformation. (The speaker went through this material quickly, so I did not have enough time to get down more notes.)

Ari indicated that the most challenging aspect of working with clients is that companies trying to adopt Hadoop are infighting. Everyone wants to claim that they solved problems with Big Data, and some want to make sure that solving Big Data does not box them out of their jobs. According to the speaker, everyone with whom he has discussed Hadoop adoption have had success with it.

---

"How VoltDB is Making Hadoop a More Productive Tool for the Enterprise"

"Boost Hadoop by deploying a high-performance in-memory database and real-time analytics into the data pipeline."

- Scott Jarr, VoltDB Cofounder

At the outset, Scott pointed out that he is going to take away three things from this event, as opposed to the last one held in New York City: (1) we are seeing a lot of commonality, thinking about things the same way, (2) we are saying there is an ecosystem, which shows a level of maturity because we recognize that there is not a single solution to all problems, and (3) velocity was talked about in terms of realtime.

The speaker indicated that he believes Big Data is a rare, transformative market, that velocity is a huge problem, that specialized databases working together are the answer, and that winning solutions will create customer value fast. Scott walked attendees through the data value chain. As data ages, the value of an individual data item decreases in value. As data ages, the value of aggregate data increases in value.

It was pointed out that sometimes we say we are not creating silos of data, but we really are even with data warehouses. But the silos are different, based on who is looking at the data, such as interactive and realtime analytics, historical reports and analytics, and exploratory analytics.

If you can solve a problem with something you have now, go ahead and solve it. Bring in Hadoop etc to solve problems that cannot be solved with what you currently have. It is not enough to capture massive amounts of data. Organizations must also sift through the data, extract information, and create actionable data.

Scott walked through a few case studies, including high throughput, relentless data feeds, fast decisions on high value data, realtime, operational analytics that present immediate visibility, and the movement of processed datat to a volume platform. (The speaker went through this material quickly, so I did not have enough time to get down more notes.)

---

"Raw Data to Real-time Decisions: Feedback Loops Using CEP and Hadoop"

"Business intelligence solutions can leverage CEP (complex event analysis) with Hadoop to make informed decisions in real-time from operational and transactional data. Tools in the Hadoop ecosystem such as Hive and Mahout allow you to perform in-depth analysis using the full fidelity raw data to modify queries to adapt to changing business needs and newly discovered patterns."

- Carmen Taglienti and Phil Young, BlueMetal Architects

BlueMetal Architects was founded in 2010 by former Microsoft executives.

The speakers began by discussing the fact that there really is a lot of overlap between the financial world and other domains, but also walked through a list of what they see as the top-10 challenges in this space: regulatory change, restructuring, globalization, data management and analytics, risk management, technological innovation, privacy and information security, business performance, customer loyalty, and talent retention.

Hadoop is great for what it does, but sometimes it is a big hammer. What is the maturity of the company considering usage of Hadoop? What are they trying to achieve?

General features of a complex event processing (CEP) system include high throughput, low latency, continuous computation (event-driven), incremental computation, temporal support (windows, ordering), pattern detection, and stream processing.

The speakers discussed Hadoop and use of realtime Microsoft StreamInsight. Hadoop excels at processing all or most historical data, and can host processing for model derivation/calibration. StreamInsight provides a CEP engine, a language interface (LINQ) for CEP queries.

Touted strengths of StreamInsight include developer support, standard LINQ (.NET 4.0+), multiple sources and joins allowed in one statement, multiple aggregates, supported projections and macros, and user-defined operators. StreamInsight is said to be resilient to stream failures, with support of check-pointing and the replaying of streams. Manageability is provided through performance counters, event log alerts etc.

StreamInsight apparently works with 6NF, and it is interesting that this normal form was mentioned because I still remember hearing about it for the first time during graduate school when it was first defined several years ago. Each stream is like a table with a defined window that can be operated upon, permitting sliding windows of time such as 3 minutes, with updates being made every minute, second etc. (The speaker went through this material quickly, so I did not have enough time to get down more notes.)

New Book Review: "The Evolution of Data Products"

New book review for  The Evolution of Data Products: The Data that Drives Products is Shifting from Overt to Covert, by Mike Loukides, O'Reilly, 2011, reposted here:

While in my opinion this short white paper sized text is not as well written as other entries within the recently published O'Reilly Strata series on Big Data, similar in size to "Data for the Public Good: How Data Can Help Citizens and Government", by Alex Howard, I am ranking this work similarly to this text and others in this series, including "Big Data Now: Current Perspectives from O'Reilly Radar" and "Planning for Big Data: A CIO's Handbook to the Changing Data Landscape", because it helps complete the current state Big Data landscape.

After discussing data products that have replaced physical products, the author contemplates the question as to how the next step should be taken, where data recedes into the background. Loukindes proposes that the next step in the evolution toward data products are those which deliver results rather than data. In addition, the author expands on the idea outlined within the subtitle of this work, that data is shifting from overt to covert, by noting that while overt products tend to depend on overt data collection, not only is data invisible in the result with covert data products, but it tends to be collected invisibly because it has to be this way.

He furthers this thought by commenting that "it has to be collected invisibly: we would not find a self-driving car satisfactory if we had to feed it with our driving history. These products are frequently built from data that's discarded because nobody knows how to use it." Within his closing thoughts, the author quotes Eric Schmidt, currently executive chairman of Google, which he said "needs to move beyond the current search format of you entering a query and getting 10 results. The ideal would be us knowing what you want before you search for it".

The author follows up by writing: "This controversial and somewhat creepy statement actually captures the next stage in data evolution. We don't want lists or spreadsheets; we don't want data as data; we want results that are in tune with our human goals and that cause the data to recede into the background. We need data products that derive their power by mashing up many sources. We need products that deliver their results in human time, rather than as batch processes run at the convenience of a computing system. And most crucially, we need data products that go beyond mere recommendation to discovery. When we have these products, we will forget that we are dealing with data. We'll just see the results, which will be aligned with our needs."

New Book Review: "Data for the Public Good"

New book review for Data for the Public Good: How Data Can Help Citizens and Government, by Alex Howard, O'Reilly, 2012, reposted here:

Similar to another recently published O'Reilly Strata series text on Big Data, "Designing Great Data Products: Inside the Drivetrain Approach, a Four-step Process for Building Data Products", in the sense that it is very succinct, in this case providing focus on data for a specific purpose, rather than solely focusing on Big Data like the other series entries I have read recently, "Big Data Now: Current Perspectives from O'Reilly Radar" and "Planning for Big Data: A CIO's Handbook to the Changing Data Landscape", this quick read is expected to be a welcome addition to the reading list of anyone beginning their investigation of government data being made available to the public (otherwise known as "open data").

After discussing the case for open data and the potential that it has for public good, Howard walks the reader through several examples of how open data is currently being used, and then addresses trends demanding special attention in this space. As a consultant who has worked for both public and private sector clients, including pro bono work for another consulting firm which provides services around open data, I especially appreciate the outline of four areas that support the investment in this space that the author shares from John Tolva, the chief technology officer for Chicago, which includes trust, accountability of the work force, business building, and urban analytics, as well as the sharing of four trends that include civic network effects, hybridized data models, personal data ownership, and smart disclosure.

And the closing thought of the author speakes volumes: "The exabyte age will bring with it the twin challenges of information overload and overconsumption, both of which will require organizations of all sizes to use the emerging toolboxes for filtering, analysis and action. To create public good from public goods - the public sector data that governments collect, the private sector data that is being collected and the social data that we generate ourselves - we will need to collectively forge new compacts that honor existing laws and visionary agreements that enable the new data science to put the data to work."

Chicago Big Data: Future of Data Processing with Apache Hadoop (August 21, 2012)

Personal notes from the Chicago Big Data meeting this past week at 1871 at Merchandise Mart, a tech hub for startups which opened just over 3 months ago, led by Bobby Evans, a developer and tech lead at Yahoo! Arun Murthy, founder and architect at Hortonworks, was planning to present with Bobby, but his flight was delayed and so was not able to make it to the meeting, and the Skype connection with him ended up being poor, so he did not speak. Bobby has been working with Arun for about 1.5 years.

• title of presentation was "Future of Data Processing with Apache Hadoop"
• Bobby gave an overview of Hadoop, discussed what YARN offers, and then looked down the road
• there are two main components to classic Hadoop MapReduce: JobTracker and TaskTracker
• a number of limitations exist in the current release of classic Hadoop MapReduce
• current limitations include scalability, a single point of failure, and restart is very tricky due to complex state
• the max cluster size is 4k nodes, with 40k max concurrent tasks
• course synchronization in JobTracker
• a failure kills all queued and running jobs, so jobs need to be resubmitted manually by users
• current limitations also include a hard partition of resources into Map and Reduce slots, with low resource utilization
• support for alternate paradigms is also lacking
• iterative applications implemented using MapReduce are 10x slower
• hacks exist for the likes MPI/graph processing
• wire-compatible protocols are lacking
• client and cluster must be of the same version
• applications and workflows cannot migrate to different clusters
• upgrading from one version of Hadoop to another is complicated: the cluster needs to be brought down, and if an upgrade for example of one node is missed, it is a disaster
• Bobby argued that what is needed is reliability, availability, utilization, wire compatibility, scalability, agility, and evolution
• enabling evolution is the ability of customers to control upgrades to the grid software stack
• with YARN, the two major functions of JobTracker are split apart: cluster resource management and application lifecycle management
• MapReduce becomes a "user-land" library, which Bobbie later defined as code that can be hacked as needed, giving the ability to innovate much more quickly
• YARN is another application that can run on top of Hadoop
• in place of the JobTracker is the resource manager, node manager, and application master
• state becomes much simpler and scalable
• the application manager is a special process that encompasses application logic, identifying where to run map tasks etc
• for the most part, recompiled MapReduce jobs will work, but some things still need to be ironed out
• YARN provides a generic resource model; the fixed partition Map and Reduce slot has been removed
• YARN right now is on the side of too generic, acting very similarly to MapReduce 1.0
• Bobby thinks YARN enables up to 10k nodes
• in addition to utilization, there are also scalability improvements
• Bobby noted that 6k machines in 2012 are greater than 12k machines in 2009
• a resource manager is coming with 2.x, with no single point of failure, saved state in ZooKeeper, and application masters restarted automatically on RM restart
• an application master has also been added, with optional failure via an application specific checkpoint
• MapReduce applications pick up where they left off via saved state in HDFS
• protocols are wire-compatible; old clients can talk to new servers, and rolling upgrades coming soon that permit failover to a new resource manager without downtime
• some customers with whom Bobby has worked are afraid to upgrade because code may break, but YARN enables pointing to different versions of the resource manager
• support for programming paradigms other than MapReduce include MPI, MAPREDUCE-3315, machine learning, iterative processing (Spark), realtime streaming (S4 and Storm coming soon), and graph processing (GIRAPH-13 coming soon)
• these programming paradigms are enabled by permitting the use of a paradigm-specific application master
• YARN is available in hadoop-0.23.2 release and hadoop-2.0.0 (alpha) release
• YARN is coming soon to hadoop-0.23.3 release (expected to rollout in September 2012) and hadoop-2.1.0 (alpha) release (also coming shortly)
• performance gains for HDFS read ops/sec of 60% more have been seen
• MapReduce sorts have decreased 14%, with 17% higher throughput
• shuffle runtime has decreased 30%, with shuffle time decreased 50%
• small jobs run with Uber AM have seen 3.5x faster word count with 27.7x fewer resources
• Uber AM checks to determine whether jobs are small, and if so runs them differently
• MapReduce currently spawns new containers for each task, and YARN 2.x will reuse task slots/containers
• while Hadoop interfaces are the same, the code underneath is a virtual rewrite
• a lot of testing has been done, including regression testing and integration testing with HBase, Pig, Hive, Oozie, etc
• YARN still has the concept of heartbeats
• right now there is a week limit for a job to run before tokens go away
• using heartbeats is really the only way to make sure a given job is alive