Erik on Software

New Book Review: "Mastering VMware vSphere 4"

Recently posted book review for Mastering VMware vSphere 4: Install and Maintain VMware vSphere 4, Create and Manage Virtual Networks, by Scott Lowe, Sybex, 2009, reposted here:

In case the potential reader of this text is wondering to what extent VMware vSphere 4 is covered, the subtitles explain a majority of the content: "Install and Maintain VMware vSphere 4" and "Create and Manage Virtual Networks". The reason this reviewer is calling out these subtitles is because some of the other reviewers here have questioned why certain aspects of the product are not covered. This text is already 650-pages in length. While the subject matter covers well installation, maintenance, and network creation (and other reviewers are correct when indicating VMware documentation and white papers are better than average when covering these topics), this book also includes a well written introduction to vSphere 4, as well as other topics such as creating and managing virtual machines, migrating and importing virtual machines, managing virtual machine resource allocation, ensuring high availability and business continuity, and monitoring performance.

This text is a well written introductory guide (meaning that although it covers some advanced topics, it is written with the beginner in mind), and although (as other reviewers have indicated) many will quickly outgrow this effort, Scott Lowe provides a great one-stop shop for getting going with vSphere 4. What the potential reader will need is highly dependent on how deeply they will get involved with the product (and technologies offered as part of the product). As a consultant architect, my reading was initially focused on the non-core portions of the book that concentrate on creating and managing virtual machines, migrating and importing virtual machines, and managing virtual machine resource allocation, although other portions of the book were later read as well. Combined with the introduction where vSphere 4 is explored and product editions and licensing are explained, this reviewer was able to begin discussing (with quick turnaround) the environment with fellow consultants and client staff after a reading of this text.

VMware documentation and white papers will need to be read in conjunction with this text once the basics are understood, although in some aspects the explanations provided in this book cannot be found elsewhere. While the bulk of the material found here is otherwise available from alternative sources, the organization that Lowe provides is superb. For example, Table 1.2 ("Overview of VMware vSphere Product Editions") is complete, listing all editions of vSphere and the options available with each, while the white paper ("VMware vSphere Pricing, Packaging and Licensing Overview") that Lowe notes at the bottom of the table, although a good reference, is fragmented in the opinion of this reviewer, and a search of the VMware website separates the edition comparison across dedicated pages for small and midsized businesses and large businesses.

One aspect of which the potential reader needs to be aware is the sparse coverage of automation in Chapter 14. Although vCenter Orchestrator, PowerCenter, PowerCLI, and ESX shell scripts are introduced, the coverage of these areas is very raw. And although the author correctly indicates that ESXi does not have a Linux-based Service Console that can be accessed, he limits his advice on porting the scripts that he shares to run within the vSphere Management Assistant (vMA). Lowe freely admits in the concluding pages of this chapter by writing that there is "much, much more to writing shell scripts than the tiny snippets I've shown you here", and anyone experienced in writing UNIX shell scripts should already know that entire books have been written on the subject. This reviewer especially appreciated closing "The Bottom Line" sections provided at the end of each chapter (and answers in Appendix A to questions intended to deepen skills and understanding by working out exercises). Well recommended to anyone needing an introduction to virtualization that is specific to VMware vSphere 4.

New Book Review: "Making EIM Work for Business"

Recently posted book review for Making Enterprise Information Management (EIM) Work for Business: A Guide to Understanding Information as an Asset, by John Ladley, Morgan Kaufmann, 2010, reposted here:

John Ladley indicates at the outset that the necessity of this text is threefold: (1) because the reader is already doing enterprise information management (EIM) to some extent but not very well, this book is intended to show how to manage information better, (2) since this book is an apology from the author following a 30-year information technology career during which time he and his peers have "absolutely done a terrible job making a business case" for addressing data and information as an asset, this book is intended to increase business understanding, and (3) because business attitudes toward data and content are stuck in a "quick-fix, I can do it myself and Information Technology are idiots" mode, this book covers the risks and challenges that can only be addressed after data and content are seen from a different perspective.

While it is not made apparent from the Table of Contents, the 30 chapters preceding the 8 appendixes are divided into two parts, the first of which (Chapters 1 through 14) is intended to provide material to sell, understand, and validate the EIM program, and the second of which (Chapters 15 through 30) is intended to delve into the methods and artifacts needed to maintain EIM and have the business manage information. Whatever the case specific to this reviewer, perhaps due to Ladley targeting executive audiences, the text started making sense beginning with chapter 6 ("A Case for the Case"), the beginning Mark Twain quote of which sums it up: "It usually takes more than three weeks to prepare a good impromptu speech." In other words, "the EIM team needs to prepare to sell, so they can go and officially sell." In the opinion of this reviewer, although the first five chapters are slow, the content in these chapters is provided as foundational material for subsequent chapters.

However, there are some bright spots within the first five chapters. For example, following a discussion on the changing role of information, the author comments that "the most common requirement I have heard from businesspeople over the last 20 years is 'Just give me all of my data. I know what to do with it.' To go out on a limb, Mr. Business Person, you do not know what to do with it. If data is an asset there must be more to managing it than just looking at a report. That is the easy part. The business person perceives an analysis result, but has no idea how to position the asset properly." Ladley furthers by posing the question "Is information (or data, or knowledge) an asset to your business?" to which he responds, "The obvious answer seems to be yes. Again, few executives would say 'no - it is not an asset and not needed for my business'. But accepting the concept and making it real mean moving from the lubricant phase of history to the fuel view of data, as well as accepting that there will be some changes in your organization - and the IT changes are the trivial ones. Accepting the definition of information as fuel and an asset means you agree to track it. Inventory it. Assign rigid and at times unmerciful accountability for its accuracy and use."

The business case for EIM is presented well in Chapter 7, beginning with a cartoon (all chapters begin with either a cartoon or a quote) that depicts two individuals sitting at a bar, with one quipping to the other, "What we need is a long-term quick fix." Humorous, but a remarkably good summary of some of the clients with which this reviewer has worked as a consultant. Corporate culture is discussed well in chapter 10, and this reviewer especially appreciated the portion of this discussion on forms of resistance within any given culture. The first part of the text ends by presenting several effective case studies, and interestingly enough the second part of the text begins by requesting that the reader return to reading the first three chapters of the book if they skipped to Chapter 15, because the same concepts discussed from an executive perspective that are reviewed from a middle management perspective are needed to increase reader understanding.

In the second part of this text, the author continues to place side-bar "Tip for Success" sections throughout the text, and this reviewer especially appreciated this aspect. Although many business books provide some type of side-bar, the high quality of the side-bars that Ladley provides are at the same level as those included in the enterprise architecture series of texts by Tom S. Graves, such as "The Service-Oriented Enterprise: Enterprise Architecture and Viable Services" and "Real Enterprise Architecture: Beyond IT to the Whole Enterprise" (see my reviews). This near-500 page work is weighty, with a liberal amount of tables and diagrams throughout, and the author's "Observation" side-bars are well placed, calling out attention to information important to the main body of material. For example, in Chapter 21 the author stresses that "The data model is not an option. If it is not done now, it will be done later. Please do not interpret the need to be conceptual or deemphasize detail as a statement of flexibility. First, we are framing the organization and agreeing on context. No other tool does this as well. Second, we are moving toward some level of detail where a standard data model can be used to spring board many technical activities, and save a good deal of time and effort."

As an architect, this reviewer especially appreciated Chapter 23 ("Architecture Part 1"), Chapter 24 ("Architecture Part 2"), and Chapter 25 ("Architecture Part 3"). Important considerations continue to be included in the presentation that Ladley provides, including the following warning: "There will be shadow IT areas that will begin about now to announce they are not going to relinquish (insert their departmental product, tool, database, or reports here), and will begin a process of complaining and/or passive aggression. This is actually good news! First of all, you have their attention. Second, you have identified additional parties requiring EIM orientation and education. Lastly, they may have an EIM element in place that can actually be leveraged." In concluding chapters, this reviewer enjoyed Table 27.2 that charts "Human Dynamics Principles" (mental, emotional, physical) alongside some of the elements of EIM (such as data governance and data quality) to show how individuals might react to EIM, as well as a discussion on GAIP (Generally Accepted Information Principles) which consists of several principles such as asset, value, going concern, due diligence, risk, and quality. Well recommended.

Countdown to 2014

In his most recent (October 25, 2010) Computerworld magazine opinion piece, Frank Hayes discusses a debt that corporate IT departments need not pay. Perhaps a majority of readers of the magazine are already familiar with "IT debt", because Hayes mentions the term multiple times within his article but never defines it. The concept is all too familiar to those of us working in the technology space, but I personally do not remember hearing the term until just a few years ago, and since that time the frequency of individuals conjuring up the term seems to have increased.

About a year-and-a-half ago, I came across a discussion of "IT debt" or "technical debt" in a book edited by Richard Monson-Haefel entitled "97 Things Every Software Architect Should Know". While I had mixed feelings about the book at the time of reading (my original review is available here), after listing some of my favorite essays I wrote the following about the contribution by Burkhardt Hufnagel:

By far "Pay Down Your Technical Debt" is the most creative of the lot and could be greatly expanded upon in a title such as that in risk management. Here is an exerpt: "So why the concern over making changes properly now versus later? It's because there's a hidden cost to making these quick and dirty fixes. For financial debts the hidden cost is called 'interest', and most anyone with a credit card knows how expensive just paying the interest on a debt can be. For technical debt, interest takes the form of instability in the system, and increased maintenance costs due to the hacked-in changes and lack of proper design, documentation, and/or tests. And, like financial interest, regular payments must be made until the original debt is repaid".

Earlier in his essay, Hufnagel mentions that such technical debt is incurred due to making a "quick and dirty" fix rather than implementing a fix the "right way" after software is already in production, and that "as the project's architect, you'll have to decide which makes more sense and then convince the decision makers to take your advice; and, as with most architectural issues, there is a tradeoff involved". The complete text of this essay is freely available with the rest of the book here, but I have already quoted half the essay, since most of the essays are limited to two pages.

Apparently, this terminology has been around for quite some time. Ward Cunningham, who reportedly developed the first wiki, is quoted as writing the following in an OOPSLA report dating back to 1992:

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.

Last year (2009), Cunningham reflected on his debt metaphor in a short video where he mentions that a lot of professionals misquote him by explaining technical debt as coming back to clean up code after writing the code poorly the first time around. Instead, Cunningham had indicated that technical debt is what is incurred after developing a solution and not properly or fully understanding the problem, and so the code would need to be revisited after greater understanding is achieved.

One can argue over whether Cunningham's original explanation can be expanded upon, but I do think that Hufnagel's limiting the concept of technical debt to released software is too limited. For example, it can easily be argued that debt can be incurred during the first iteration of a four iteration software development cycle even though the associated software has not yet been released to production.

Regardless, however, it appears that Gartner estimates outdated software like Internet Explorer 6 (IE6) has created a "global IT debt" of $500 billion that is expected to rise to $1 trillion by 2015. The closest Hayes gets to a definition of this debt is the "cost to update old apps so they're shiny and new and fully supported". As an architect, I hope that this thinking is severely limited to a small group of CIOs chasing shiny objects.

The point of Hayes' latest opinion piece, however, is to actually argue that it is worth defaulting on some of this technical debt. He goes on to explain:

Look, how did those IE6-dependent users get into that position? They bought into the last big paradigm of software development: the Web browser as platform. Remember? All our apps would be built with a Web front end. Microsoft thought that was a fine idea and built a lot of non-standard features to lock customers in. Vendors and corporate IT shops used those features to make their Web apps just like regular Windows apps - and users were locked in for good.

How will these users get out of this mess? When the time (and budget) is right, they'll jumpon the next big software development paradigm. They'll buy or build fresh, junk the old stuff completely and go about their business.

IT debt? They'll walk away from it. And why not? Those apps built on IE6 may be creaky, ugly and nonstandard, but they still work. And Microsoft may hate its nine-year-old miscalculation, but it has promised to keep supporting IE6 until 2014.

Besides, how much sense would it make to patch up those old clunkers or migrate them to newer browsers? None at all. Those old apps were designed for business needs from as much as a decade ago. Technical upgrades won't solve that. Building or buying new apps will - and doing that will cost the same, whether the apps they're replacing are up to date or ready to collapse.

The sensible thing is to keep running those IE6-dependent Web apps until they can be replaced with new apps that run from the cloud or on tablets or using whatever the hot new paradigm is.

The question in my mind here is whether the debt to which Hayes alludes refers to financial cost only. If so, he might have a point. But reading between the lines, I think the cost goes far beyond the financial to include opportunity cost. Hayes actually indirectly mentions opportunity cost by referring to continued use of old applications that address old business needs. He refers to present development cost, but not the cost of lost opportunity incurred over all the years since the associated software lost its relevance. That is the true technical debt.

(The quote that opens Hayes' essay is a classic: "We have to use IE6," a contractor at one telco told me recently. "We have all these Web applications that won't run on a browser that isn't broken.")

SpringOne 2GX 2010 Day 2: Keynote

After returning to work Thursday and catching up on some sleep following my attendance of the SpringOne 2GX 2010 conference held at The Westin Lombard Yorktown Center in the heart of the Lombard / Oak Brook, Illinois business corridor, the following are some notes taken following dinner during the second keynote address led by Adrian Colyer, SpringSource CTO and leader of open source project AspectJ. (Notes taken during the first keynote address on Day 1 of the conference can be found here, and the complete session schedule for the week (Tuesday evening through Friday afternoon) is located here.)

Along with Colyer, a few other individuals spoke directly to specific products within and around the SpringSource lineup. This second keynote ran much more smoothly than the first, and all of the software demonstrations appeared to have worked, unlike during the first where network connectivlty periodically stalled (or stopped) demonstrations. In contrast to Rod Johnson, rather than working up to a theme Colyer announced the theme of his keynote to be "Happy Hour at the Microbrewery", a theme which is of course meant to be humorous, although giving no indication of keynote content.

Colyer opened up his keynote by displaying an animation that showed the history of Spring Framework 3.0 by graphically depicting Subversion commits over time using Spring Batch, Spring Integration, and WebSocket API (and incidentally, this was the only or one of the only mentions of Subversion the entire evening; all other mentions of version control focused on Git).

This demonstration was then immediately followed up by Colyer with a question: "How can I move applications through the development, testing, and production environments easier?" He then presented "Spring Travel", the Spring MVC and Spring Web Flow reference application powered by Spring on GAE (Google App Engine) that can be found here, followed by essentially the same application hosted on VMforce, which combines the Java programming language, the Spring Framework, and the Force.com cloud platform. This second reference application can be found here.

Spring Framework 3.1 introduces "environments" and "profiles" which are intended to enable movement between environments more efficiently and can be activated via web.xml, systems properties, or environment variables. One or more profiles can be set as default. (In the midst of this discussion, Colyer announced that Spring Framework 3.0.5 was released earlier in the day.) Essentially, what was being sold here is environment-based property injection.

In addition, Spring Framework 3.1 introduces a caching abstraction via the @Cacheable annotation for incorporating pluggable caching strategies into applications. The @Cacheable annotation uses Spring AOP and AspectJ under the covers, and can be invoked using

<cache:annotation-driven ... />

Caching can be provided via the @Cacheable annotation at the local node level, but distributed caching is also available. The following swappable distributed caches are built in: Ehcache, Coherence, GigaSpaces, and GemFire (recently acquired by SpringSource). Colyer offered a simple description of distributed caching: "horizontally scalable data fabric" that enables "data shuffling" between two WANs.

Colyer followed the caching discussion by what he described as an "enterprise integration fabric", horizontally scalable messaging fabric that enables WAN integration via AMQP (a standard that provides messaging interoperability via a wire-level protocol rather than at the API level like JMS). RabbitMQ implements the AMQP standard and was recently acquired by SpringSource, and emphasized that all it takes to install the software using the Brew package manager is

> brew install rabbitmq

Spring Rabbit is an implementation of RabbitMQ and is the first implementation of Spring AMQP. Under the covers, calls by Spring Erlang are made to Erlang from Java (RabbitMQ was developed with Erlang). I am sure you have noticed by now that a number of technologies are being brought under the Spring fold. In addition, some of the technologies mentioned during the first keynote were revisited by Colyer during his discussion, albeit at a more technical level.

Colyer extended the discussion of NoSQL started during the first keynote, by first reviewing some of the products in this space - Apache Cassandra, Redis, MongoDB, Apache CouchDB, Neo4j, GemFire, BigTable, Apache Hadoop, JCR, and Riak - Graeme Rocher, project lead and Grails co-founder, was introduced to present NoSQL with Groovy and Grails (SpringSource acquired G2One, the company behind Groovy and Grails, in 2008).

Rocher opened with an answer to his own question as to how Grails users can work with data stores (GORM), and offered a discussion on GORM for NoSQL. During his demonstration of using GORM for Redis, Rocher focused on its ease of use:

• start up the Redis server
• create a Grails project using SpringSource Tool Suite (STS)
• open plugin manager
• uninstall Hibernate plugin
• install Redis plugin
• create domain class
• load Grails console
• start executing methods

Following the demonstration, Rocher indicated that a lot of the NoSQL products address specific problems like scale, and announced that GORM for GemFire was being made available for immediate release, a demonstration of which followed where he again emphasized ease of use. On the tails of the GORM for Redis demonstration, he executed the following steps:

• install GemFire plugin to STS
• load Grails console
• kill Redis server
• start up GemFire

> grails gemfire-generate

> grails start-cache-server

• start up two Grails consoles
• start executing methods from either console

Rocher indicated that each of the Grails consoles were running in a separate JVM. In addition, he emphasized that support for continuous querying was added for GemFire. After turning this feature on during the demonstration, he showed how results for a continuously executed query were displayed in the other console, which has implications for a wide variety of usage.

Colyer returned by introducing the Spring Data project, a programming model for productivity and consistency across a wide range of NoSQL products, the M1 release of which is planned for 4Q2010. During this introduction, he provided a preview for a CouchDB demonstration to be presented in a conference session later in the week that will show offline data access and synchronization with continuous bi-directional replication between an Android mobile phone and a Spring application server.

Keith Donald, SpringSource principal and founding partner, continued a discussion of Greenhouse started during the first keynote by talking about Spring Social and Spring Mobile. Donald discussed Greenhouse as an OAuth (open standard for authorization) provider, using both an API key (permitting another application to connect to you while running the Greenhouse client from a mobile phone with the Safari browser) and a "secret" key. Access tokens are stored within the browser database, including both the API key and secret key that are both encrypted (the encryption method was not discussed).

Permanent OAuth connections can be made via Spring Security. The first milestone of Spring Social is planned for M1 release later this week. In addition, the M1 release of Spring Mobile is also planned for this week. Spring Mobile provides automatic conditional view rendering depending on the device by which an application is accessed, and from one line of code these different layouts are effective across all pages of the application.

Jon Travis, Spring Insight architect and co-founder, took the stage to demonstrate Spring Insight, which provides monitoring of requests across an entire VMware cluster in real-time, including the ability to easily drill down into requests such as JDBC. During the demonstration, Travis commented that "the concept of the cloud is really open to debate", mentioning that instances of tc Server for example are really "just transient objects that can come and go".

(Note that during the conference, I had a chance to chat with a few SpringSource (and coincidentally ex-SpringSource) representatives during meals, and one individual from SpringSource replied "really not much" to my question as to what makes tc Server different than Apache Tomcat.)

SpringOne 2GX 2010 Day 2:Virtualizing Your Java Applications

Virtualizing Your Java Applications: Best Practices (Justin Murray)

Emerging Technology Track 8:30 AM - 10:00 AM

---

From the conference materials:

In this session, you will learn the key things to consider when you take your application to a virtualization platform, in particular to VMware ESX. The choice of configured memory sizes, virtual CPUs, timekeeping strategies and general virtual machine monitoring will be explored. Best practices will be detailed in each section. An example performance test project will also be shown.

Justin Murray holds a technical alliances role at VMware, where he has worked for three years. His main work at VMware is in helping ISV partners to test, prove and optimize performance of their products on the VMware vSphere platform. Justin also helps VMware's customers with virtualizing their business applications and infrastructure and advises them on best practice approaches to virtualizing Java-based applications in particular. Previous to joining VMware, Justin worked at HP on management software and on Java on HP-UX.

---

From my personal notes:

• Use of VMware ESX (not VMware ESXi or VMware Server) is the focus during this presentation.
• Technically, no changes are needed when moving Java to ESX.
• This session shares insights based on real world customer scenarios by Justin Murray from VMware.
• There are only 7 best practices here, all of which are well documented in the VMware white paper on this subject, although not everything presented in this session exists in the white paper.
• Only about 25% of attendees (by raise of hands) appear to have already started virtualizing Java applications, and a small portion are currently considering doing so.
• The Hypervisor gets out of your way most of the time, passing down Java instructions directly to the JVM most of the time.
• If you follow these best practices, getting within 5% to 10% of the performance achieved without VMware is obtainable.
• There is a large wave of applications currently being migrated to VMware in the industry.
• The classic use case of virtualization is server consolidation.
• The ratio of physical to virtual servers is typically 5, although this number can be much higher.
• About 90% of servers typically run at about 14% or lower of CPU capability.
• Running multiple Java instances within a virtual machine is a better way to use the additional power available while at the same time providing isolation across applications.
• It is much easier to isolate performance problems when using virtual machines.
• Isolation at the OS and JVM layers enables greater flexibility for patch management.
• Rapid provisioning can be performed by storing a "golden" virtual machine in a library and copying for other instances.
• VMware HA (High Availability), DRS (Distributed Resource Scheduler), FT (Fault Tolerance).
• It is recommended that only 1 to 3 JVMs be running in each virtual machine, although Murray has seen customers run dozens per virtual machine.
• Java is probably the easiest platform to move from physical to virtual.
• The Hypervisor is essentially a customized OS that specializes in hosting virtual machines.
• There are 3 layers of memory that support Java: physical server memory, virtual machine memory, and JVM memory.
• Do not overcommit your ESX server.
• The ESX Hypervisor has no idea that memory references are associated with Java.
• JVM garbage collection acts the same way when running on a virtual server.
• Transparent page sharing enables memory overcommitting, but Murray advises to stay out of that scenario when using Java.
• Memory management between guest physical memory and machine memory is now handled by the latest hardware.
• Use the "very, very latest" hardware for Java systems - right now the Intel 5500 and the AMD Shanghai - typical results demonstrate 25% performance increase.
• Extended page tables (EPT) are available with both the Intel 5500 and AMD Shanghai.
• The motivation of ESX is to share as much as possible.
• Switch off the page sharing algorithm between virtual machines.
• Ballooning enables emergency retrieval of memory, but VMware has no idea that it is affecting the JVM.
• ESX has a SWAP file for each virtual machine, but we do not want swapping to occur because of resultant performance degradation.
• The following features need to be turned off: page sharing, ballooning, and swapping.
• How big should the virtual machine be? The virtual machine needs to be sized to leave adequate space for the Java heap and other memory demands of the JVM code, for any concurrently executing process that needs memory from the same guest OS to prevent swapping in the guest OS.
• The memory "Reservation" value in the vCenter client needs to be set to the size of the active memory for the virtual machine, but a problem is that people often do not trust themselves when setting this value, so use the configured memory if you are in any doubt about "active memory".
• Using large memory pages provides up to a 20% performance boost.
• Page sharing between virtual machines is based on content - it is possible that using the same guest OS for multiple virtual machines will increase this capability.
• A single Java thread executes on a vCPU at any given time.
• What is the right number of vCPUs to assign to a given Java process? Perhaps 1 vCPU is all that is needed. It is possible that Java software being investigated is not as multithreaded as one might think. The majority of Java threads should not be in the running state, but in the waiting state or the ready to run state.
• It is not possible to look at the static threads and determine how many vCPUs should be used.
• Generally speaking, it is alright to have more vCPUs than physical CPUs, provided that they are not running at 100% all of the time.
• It is feasible to have overcommitted vCPUs, not overcommitted physical CPUs.
• Run load testing with varying amounts of vCPUs to determine the effect on a Java application - start with the lowest number of vCPUs that is practical.
• More vCPUs is not always better.
• Set limits on real CPU power when maximizing utilization of ESX host with multiple vCPU virtual machines.
• Use vCenter and "esxtop" command. The level of granularity is finer with esxtop - about every 3 seconds as opposed to every 10 to 15 seconds using vCenter.
• Be careful with the amount of I/O logging.
• The "esxtop" command is modeled after the traditional Linux "top" command.
• Use "esxtop" with "netstat -a" to determine any network related bottlenecks.
• Use "esxtop -n".
• Synchronize between ESX host and virtual guest OS with external NTP source for Linux or w32time for Windows.
• Lower the clock interrupt rate on Linux machines by using a more modern kernel if possible.
• Use as few vCPUs as are practical for your application.
• Kernels for best timekeeping: for ESX 3.5 use SLES 10 SP2 (use VMI for 32 bit VMs), RHEL 5.4. (Note: missed recommendations for ESX 4.0.)
• Use your actual application code in sizing exercises rather than representative code or subsets of what you think represents your code.
• For easier monitoring and load balancing, use one JVM or a small number of JVMs per virtual machine.
• Use "esxtop" output data to see where resource consumption occurs in your Java application.
• What you lose when having more than one JVM per virtual machine is that when you use vMotion you will not know exactly which JVMs are being moved.
• White paper will be updated again in the next few months.
• A tool called "Capacity Planner" can be run against virtual machines to gather statistics to determine how many can be virtualized without even thinking.
• Capacity Planner monitors context switches per second, disk I/O, memory consumption, and other concurrent process besides Java consumption.
• Use the latest versions of ESX/vSphere and hardware with virtualization assist if possible.
• Latest ESX 4.1 combined with latest hardware can provide as little as 0% overhead - the latest can handle 200k IOPS greater than 95% of Oracle databases in the world.
• In some cases, virtualized platforms have even greater performance than physical counterparts.
• Performance brief white paper.

SpringOne 2GX 2010 Day 1: Keynote

Just returned from the first day of the SpringOne 2GX 2010 conference held at The Westin Lombard Yorktown Center in the heart of the Lombard / Oak Brook, Illinois business corridor. The complete session schedule for the week (Tuesday evening through Friday afternoon) is located here.

From the conference materials:

SpringOne 2GX is a one-of-a-kind conference for application developers, solution architects, web operations and IT teams who develop, deploy and manage business applications. This is the most important Java event of 2010, especially for anyone using Spring technologies, Groovy & Grails, or Tomcat. Whether you're building and running mission-critical business applications or designing the next killer cloud application, SpringOne 2GX will keep you up to date with the latest enterprise technology.

Featuring over 100 presentations delivered by development leads and published authors on the Spring, Groovy/Grails, Tomcat and Cloud technology today, it's the ideal place to obtain critical skills to help you build, run and manage tomorrow's software solutions. Plus you'll meet and learn from open source leaders who drive innovation for these technologies.

There exist several different tracks of sessions during the conference this week, and several work colleagues and I will be attending sessions during various periods of time that interest us both professionally and personally. For example, I am personally interested in sessions pertaining to moving applications to VMware products due to the virtualization work I am performing at my current client.

This evening I attended the dinner and follow-up keynote address led by Rod Johnson, founder of the Spring Framework. Johnson is currently General Manager of the SpringSource Division of VMware (SpringSource was acquired by VMware in 2009). Along with Johnson, a few other individuals spoke directly to specific products within and around the SpringSource lineup to provide a preview of the sessions.

According to Johnson, there are twice the number in attendance this year than last year. The three platinum sponsors this year are SalesForce, Google, and Accenture. During his presentation, Johnson commented through a series of slides that it was especially difficult to arrive at a theme for his keynote address, due to the high amount of activity in the last year and a 60-75 minute limit.

Of course, Johnson was working up to a theme that he had already chosen, but the short-lived suspense was a winning formula for the time he had available to do so. In working up to the theme, Johnson indicated that the keynote could have been about any of a number of topics: VMware vFabric Cloud Application Platform, GemStone GemFire Enterprise Data Fabric, the software associated with the recently announced UID for 1.2 billion Indians that utilizes RabbitMQ, Spring Insight in tc Server and integrated within SpringSource Tool Suite, Grails, GORM (which is being extended to support non-relational databases, resulting in a possible name change), Spring Framework 3.1, Spring Web Flow 3.0, and / or Spring Integration 2.0.

However, in searching for a topic, Johnson came upon the realization that the Spring Framework is almost 10 years old after discovering that RequestHandledEvent is the oldest class in the Spring Framework, dating to January 17, 2001. Because of this, Johnson arrived at the following theme: "Spring: Toward the Second Decade".

Johnson repeated the refrain throughout the opening keynote that Spring core values have always been portability, productivity, and innovation. Portability in the past essentially pointed to choice of application server to which to deploy, and over time it became clear that Apache Tomcat was the most popular choice. However, Johnson indicated that we need to view portability in the present day more broadly.

Spring abstracts business logic away from the deployment environment. According to Johnson, we need to think about increase in productivity in terms of both the development stack as well as the process that is used for that development (Grails and Spring Roo were sited during this part of the discussion). We need to address how developers work, to be effective throughout the application life cycle.

Lead developer of SpringSource Tool Suite (STS), Christian Dupuis, followed this discussion with a demonstration of Grails development using STS, to which he commented that STS provides Grails development efficiency far greater than what was available two years ago. Apple afficianados might be interested in knowing that yes, he used a MacBook for the demonstration.

Johnson then discussed two recent developments available with Spring Roo 1.1: database reverse engineering and GWT development through SpringSource collaboration with Google (to which Johnson also commented that there are present many Google developers at the conference). The database reverse engineering made available uses only standard (i.e. non-implementation specific) JPA annotations, and supports round-tripping so that only pertinent changes are made to the code when the database changes.

After a brief mention of the Spring Payment Services project, Johnson reiterated that innovation has always been important to Spring, and indicated that "the world of IT is fundamentally changing". Requirements have changed because among other things it is not about the traditional browser anymore, and there are are higher expectations with responsiveness. And tying into the underlying marketing theme of the conference, Johnson quipped that "Spring is the Java programming model for the Cloud".

A demonstration of SpringSource Greenhouse, apparently a reference application driven by Spring products, was then given, although a portion of the demonstration did not work due to lack of network connectivity. According to the presenter, the entire Greenhouse stack consists of Apache licensed software, and the client is now available via the iPhone App Store.

After the demonstration, Johnson returned to discuss the support for NoSQL databases by Spring Data, and provided a reminder that NoSQL no longer means "no SQL" but instead "not only SQL", to which I was reminded of the claim by Sun that JDBC officially does not stand for "Java Database Connectivity". Spring Data provides support, for example, for different types of data other than that residing in relational databases, such as key-value, column / BigTable, document, and graph.

Johnson discussed Neo4j partially because of the his code contributions to the product. He briefly touched on the fact that there exists trade-offs between different types of data stores, mentioned that Neo4j is "very high performance", and that its sweet spot revolves around social networking, bioinformatics, fraud detection, and MDM (although no detail was provided behind any of these applications).

The CEO of Tasktop Technologies discussed the missing links that his product provides, as well as the Code2Cloud service to be provided by VMware that link the developer desktop (Grails, Roo, STS), software development lifecycle (source control, issue tracking, builds, continuous integration), and cloud deployment. The CEO indicated that the time spent configuring software can be high, and although it might be fun at first, developer time can be better spent delivering business value rather than worrying about infrastructure. The GA for Code2Cloud is planned for 2011, with a forthcoming developer preview.

Johnson concluded by communicating to attendees that while Spring is tackling new challenges in a changing IT world, the mission of Spring remains unchanged.

The following are some additional photos I took during dinner while the conference halls were empty.

Technology == Salvation ?

Due to service disruption with my print subscription to The Wall Street Journal (WSJ) this past week, I was not aware of the interview with Peter Thiel entitled "Technology = Salvation" by Holman W. Jenkins Jr. in last weekend's edition until today. Thiel was one of the co-founders of PayPal and an early investor in Facebook, and now serves as president of Clarium Capital, a global macro hedge fund that he founded in 1998.

Thiel offers some interesting perspectives in the interview of interest to the technology community. Some excerpts:

• Except for computers and the Internet, the idea that we're experiencing rapid technological progress is a myth.
• "People don't want to believe that technology is broken...Pharmaceuticals, robotics, artificial intelligence, nanotechnology - all these areas where the progress has been a lot more limited than people think. And the question is why."
• "The big variable that [Bernanke, Geithner, and Summers] are betting on is that there's all this technological progress happening in the background. And if that's wrong, it's just not going to work. You will not get this incredible, self-sustaining recovery."
• The great exception is information technology, whose rapid advance is no fluke: "So far computers and the Internet have been the one sector immune from excessive regulation."
• Some companies and countries will do better than others. "In China and India," he says, "there's no need for any innovation. Their business model for the next 20 years is copy the West."
• The West, he says, needs to do "new things." Innovation, he says, comes from a "frontier" culture, a culture of "exceptionalism," where "people expect to do exceptional things" - in our world, still an almost uniquely American characteristic, and one we're losing.

The fact that so many comments were posted in response to the online version of this interview in such a short time period demonstrates that there is a lot of interest here, but the substance of most of the comments is unfortunately lacking (probably due in part to the text of the interview being available to the public without a WSJ subscription).

As someone who studied economics as an undergraduate, I agree with Thiel that politics plays a heavy role in how the U.S. economy will play out in coming years. It is difficult enough to predict the future when the markets are able to work things out on their own, but when the government gets involved it is nearly impossible. Of course, I use the term "predict" loosely here - what I am referring to is estimating event probability.

And while I was not overly impressed with Michael E. Raynor's "The Strategy Paradox: Why Committing to Success Leads to Failure (And What to Do About It)", in my review of the book I noted the following:

Any future strategy, according to the author's explanation of the Strategy Paradox, may fail because the future cannot be predicted: "strategies with the greatest possibility of success also have the greatest possibility of failure". Planning and executing well a strategy does not necessarily increase the chances of success. Instead, companies should hedge their bets by developing practical strategies based on multiple options that respond to the different requirements of several possible futures rather than commitments on singular strategies.

There exists no public talk regarding the development and execution of multiple strategies for the U.S. economy. At present, the driving singular strategy is offshoring work in order to gain very short term profits. Offshoring complete industries will render the U.S. heavily dependent on other countries until the strategy is changed, and it is becoming increasingly clear that this dependence is not feared.

I agree with Thiel that the unfortunate business model that has been the main driver for growth in economies such as China is to "copy the West". The U.S. rebuilding of Japan and Germany following the second world war acted as a powerful catalyst for economic growth, but that growth would have been less probable if it were not for the intelligence and work ethic inherent in those countries.

While it might be true that China is "coming back" rather than "rising up" (as Chinese economist Dr. Peter Zhao Xiao noted in the recent conference I attended), the economic progress in China over the last few decades would have been unlikely if it were not for the vast investments from the West. And one of the resultant problems is that the accompanying knowledge and skills have followed these investments.

As a consultant, I need to be able to keep up with advancements in my field and reinvent myself on a continual basis in order to remain viable. Every year I experience periods of time where I am running in order to learn new skills in order to provide what is best for my clients. Knowledge transfer and mentoring are typical products by design during these periods of time, but the lifetime of each growth spurt is limited.

Applying this running analogy to the U.S. as a whole, there is a mathematical limit to how fast these sprints can be executed, and how long they can endure, when a sizeable part of the end products is also the ability to do the work itself without regard to whether subsequent sprints continue to contain innovation (and there are legal implications when products and processes are copied).

While my field is fortunate in that innovation still occurs, and there continues to be opportunity to advise clients and develop solutions, it is unknown what will happen if U.S. innovation in my field ever ceases. In my opinion, part of the country will be lost.

New Book Review: "Rules to Break & Laws to Follow"

New book review for Rules to Break & Laws to Follow: How Your Business Can Beat the Crisis of Short-Termism (Microsoft Executive Leadership Series), by Don Peppers & Martha Rogers, PhD, John Wiley & Sons, 2008, reposted here:

 

Purchase of this text by this reviewer followed on the tails of "The Strategy Paradox: Why Committing to Success Leads to Failure (and What to Do About It)" by Michael E. Raynor (see my review) and "Strategy Safari: A Guided Tour through the Wilds of Strategic Management" by Henry Mintzberg, Bruce Ahlstrand, and Joseph Lampel (see my review). While initial interest in this work by Peppers and Rogers was due to its seeming focus on the strategic versus the tactical, however, potential readers should note that the authors instead investigate and present general principles that any firm should follow.

At the outset, this text does not beat around the bush. For example, in the first chapter the authors respond to the following first three rules to break by writing "No, no, no. No to all the above. These Rules to Break are really just assumptions about how business works, at the most basic level. They probably aren't written down anywhere in your strategy document, but they have almost certainly backed up your thinking and your company's actions for as long as you remember.": (1) "the best measure of success for your business is current sales and profit, (2) "with the right sales and marketing effort, you can always get more customers", and (3) "company value is created by offering differentiated products and services".

The authors further this instruction by pointing out that all of these assumptions are wrong. "If you operate according to these false assumptions, not only will your business fail to create much value, but you'll also soon find yourself trapped in a Crisis of Short-Termism. Everything you do will be so furiously centered on making today's numbers that you will become increasingly blinded to everything else. Business swept up by this crisis find that even as they try to do the right thing for their shareholders, they end up destroying value rather than creating it. So while these Rules to Break may look no more dangerous than ordinary common sense, in truth they're deadly."

According to the authors, executives need to change their mental models of what it means to succeed. The new model that Peppers and Rogers is based on the understanding that long-term business success relies on customer success as a prerequisite, and that employee trust of their employer precedes employee drive to earn customer trust. Interestingly, even though new technologies are presented as making possible sophisticated analytics, subverting the power of hierarchies, connecting consumers electronically, and undermining the advantages of new products, the authors state that it is customer trust that will provide a guideline for action even if financial metrics are not yet at the needed level of sophistication.

In the opinion of this reviewer, the third chapter entitled "Customers Are a Scarce Resource" is one of the best in the book, and if the potential reader does not have enough time to read all the chapters, they need to read this one. In this chapter, the concept of customer lifetime value (LTV) is discussed. "New customer acquisition has always been the quintessential goal of traditional marketing and is something often trumpeted to shareholders. But raising your customers' lifetime values is just as important, because over and above whatever profit you are earning currently, increasing your customers' LTVs increases the value of your business."

The authors continue by stating that the philosophy of some businesses is "based on the belief that as long as its sales and marketing effort is effective, it can always acquire more customers from somewhere. But this is a false assumption - a Rule to Break. Instead, to make the right decisions as a business, you must always take into consideration the population of customers and prospective customers truly available to you. After considering the whole population of customers and prospects, your job is to employ that population to create the most possible value for your firm. Because customers are scarcer than other resources, using up customers is more costly than using up other resources."

The fifth chapter was also especially enjoyed by this reviewer, where in discussing how to increase the value of one's business the authors present the concepts of "customer equity" and "Return on Customer" (ROC) that follow a discussion of ROI. The examples and accompanying tables in the third and fifth chapters are very well done. The thoughts that the authors share on innovation in the tenth and eleventh chapters were also appreciated, where the production of new ideas is said to be more important than making money from every new idea, and where the need to differentiate between "wise failures" and "fiasco failures" is said to be an important distinction, reminding this reviewer of a quote by James Cameron, who was recently quoted as saying "If you set your goals ridiculously high and it's a failure, you will fail above everyone else's success".

Google Analytics: 12 Months

Google Analytics was set up for this blog one year ago today. Take a look at the posts I submitted over the past year to see how traffic has grown in the past 12 months. Since the last related post 2 months ago, the number of countries of origin has increased by 8 to reach 71. Visitors from the top 10 countries of origin came from the United States (53.3%), Canada (4.4%), the United Kingdom (3.7%), Germany (3.3%), India (3.0%), the Netherlands (2.5%), Slovenia (2.1%), France (2.1%), Australia (1.8%), and China (1.5%).

This blog entered a third phase in its life cycle in late-July of this year, when a higher number of visits began to occur on a sustained basis. See the Google Analytics plot of daily visits over the past year that I annotated below. Prior to the current phase, this blog initially experienced a stagnant phase when few visits occurred, as well as a second phase when frequency of visits was all over the map (including days where frequency of visits was extremely limited). The most extreme outliers in this plot are the result of one of my book reviews being tweeted, and a popular series of conference notes that I wrote. 

Sweden and Brazil have dropped off the top 10 visiting countries to make room for Slovenia and China. To enter the top 10 in such a short period of time is quite a feat. Drilling down in the analytics, it is apparent that the vast majority of visits from Slovenia were to view my book review of "Amplifying Effectiveness: Collected Essays", edited by Gerald M. Weinberg, James Bach, and Naomi Karten, and the vast majority of visits from China came to view my notes from Global Leadership Summit 2010 (day 1 and day 2), especially my post on the session led by economist Dr. Peter Zhao Xiao.

The top 10 visiting city over the past year was Ljubljana (Slovenia), followed by New York, New York (the United States), San Francisco, California (the United States), London (England), Burbank, California (the United States), Beijing (China), Cambridge, Massachusetts (the United States), Chicago, Illinois (the United States), Los Angeles, California (the United States), and Singapore.

Venezuela continues to lead for the number of pages per visit (8.00), followed by Lithuania (7.00), Bulgaria (4.00), Portugal (3.50), Argentina (2.83), Hungary (2.50), Denmark (2.33), Austria (2.33), Norway (2.17), and France (2.15). Switzerland, Mexico, Greece, and Japan have dropped off the list of top 10 countries according to pages per visit to make room for Lithuania, Bulgaria, Austria, and France.

Visitors from the 71 countries of origin came from 686 cities. Because looking at statistics at the city level involves drilling down in the data to a lower level, it is expected that the top 10 visiting cities will continue to remain in flux. Orthez, France continues to lead as a statistical outlier for the number of pages per visit (17.00), followed by Blue Bell, Pennsylvania (15.00), Caracas, Venezuela (8.00), Shibuya, Japan (8.00), Bartlesville, Oklahoma (8.00), Frisco, Texas (7.00), Southlake, Texas (7.00), Vilnius, Lithuania (7.00), Funchal, Portugal (7.00), and Oakton, Virginia (7.00).

While a cursory glance at the second map overlay above (pages per visit) seems similar to the first (visits), there exist some subtle differences. You might notice that the first map overlay does not show data points across 71 countries. This is due to the fact that data points do not display until a certain number of visits are reached. This might reduce clutter, but I would much prefer that all data points be plotted. The second map overlay compensates somewhat because it shows pages per visit. For example, a data point for Lagos, Nigeria is displayed whereas in the first map overlay activity from that location is not represented.

In a recent post, I commented on the popularity of my posts on running, which I find somewhat humorous due to the subtitle of this blog ("musings on business and technology"). Here are links to the top 10 posts, in order of number of visits: Anima Sana In Corpore Sano, New Book Review: "Manage Your Project Portfolio", Liquibase in the Enterprise - Part 2, Liquibase in the Enterprise - Part 1, Switching from ASICS to Brooks, New Book Review: "Amplifying Your Effectiveness", New Book Review: "Employees First, Customers Second", Global Leadership Summit 2010 - Day 1: Dr. Peter Zhao Xiao, Global Leadership Summit 2010 - Day 1, and New Book Review: "Strategy Safari".