Speaking at RedHat Summit / JBoss World 2010

Posted on June 2, 2010 by Rob H

I’ve been enlisted by my employer, Dell, to speak about cloud software architecture JBoss World 2010 in Boston the week of June 21st.

My talk will expand on the “RAIN” posts that I’ve written before with some practical examples on our we are using Joyent to create applications using these models.

Here’s the abstract:

The need for hyper-scale and the lack of SLAs on public clouds has forced architects to stripe their applications across multiple servers. Similar to disk RAID striping, application striping creates redundancy using an array of inexpensive nodes (RAIN). This technique enables applications to have dramatic performance bursts while improving fault tolerance and reducing costs.

In this session, Rob will review how to use JBoss Enterprise Middleware to create a RAIN configuration using technologies available through the Dell Cloud Solution for Web Applications and on Joyent public cloud hosting. He will review the essential role of the virtual load balancer using Zeus ZXTM. Rob will also show specific architectures that can be implemented quickly and explain how ZXTM can deliver scale-out ready SQL read-write splitting without recoding.

Tippy Towers

Posted on May 6, 2010 by Rob H

or “Synergistic Nondistruptive Integration”

Back in my Space Suit Architect days, I was a fan of “green field” or second system redesign. It made sense that we should use our hard earned knowledge of the application domain to “build the product right this time instead of rushed crapware.” Since then, I’ve come to appreciate a more pragmatic approach that I call “Tippy Towers.”

Let’s get specific with an example…

My company, BEware, has a shipping product, Alpha, that works pretty well even though it’s all written in this obsolete language AWFOL-88. Yeah, it’s got problems scaling, only runs on Vista, and it’s hard to add new features; however, it’s earning revenue and the sales team can sell it.

The engineering team is really hot to recreate Alpha as a new product, Omega, using Ruby on Rails (yeah, baby!) and marketing is behind the move because they’re told it’s the only way to add features. Side note: in my experience, these claims are always exaggerated but generally true (look for a future post about working out of order).

In a green field approach, everyone agrees to stop working on Alpha and focus all work on Omega. The company enters the tunnel of death and hopefully dies of starvation before a buggy v1.0 version Omega ships to the companies few remaining customers.

Here’s the plan. Since we’re astronauts, our design is to build the new hotness Omega first and the later bring over the boring old features of Alpha (like an installer). As good Agile engineers, we’ll build the most critical use cases into Omega first. We’ll identify early customers that only need a subset of the Alpha features. Since these are a known subset and core features, we quickly build some kick-ass next-generation hypen-inducing wonderware.

The early trial customers love the stripped down Omega product and start demanding more of the Alpha features. Now the Alpha tower begins to tip over because we need Omega to duplicate the all features that are working adequately in Alpha. Customers can do a comparison between Alpha and Omega and conversations start saying “I would buy Omega but I need to green cross fade accounting API from Alpha.”

At this point, our Alpha tower is free falling. Omega has enough features that it is hurting the sales for Alpha while customers wait for Omega. Worse, abandoned Alpha is falling behind in the market. At this point, we’re going to spend many months porting working Alpha features from Omega. This is not value added work, its quality optional crisis management where we rush Omega to be “feature complete” ASAP.

Our alternative is to see Alpha as a tippy tower that we want to keep standing. Each feature we pull from Alpha to Omega can makes it more unstable. It takes more effort to support both; however, if Omega can leverage Alpha and vice versa then we have the luxury to focus on new features and better designs.

In a Tippy Tower mindset, Alpha is valuable. The existing code base is field proven, generating revenue, and shipping. Even if Omega holds promise, it’s worth our time to keep Alpha selling. It may slow down Omega, but it ultimately gives the company more time to get Omega right.

Let’s take the Tippy Tower concept a farther. I argue that Alpha and Omega should be designed to co-exist. For example, Omega’s new web interface could be displayed on a form in Alpha’s UI. It may take some time to enable a browser in Alpha, but Alpha users would have immediate access to Omega’s new capability. In this case, we’ve done work that helps create Omega and also added uplift to our shipping product.

In an ideal Tippy Tower case, those legacy Alpha features just stay in the Alpha code base and die a natural death because Omega replaced them with a better paradigm. The fact that we keep Alpha from tipping over kept us from having to port them and kept us from creating more rushed crapware.

One last note: sales people love this approach – they always have something to sell and customers to reference.

Just Striping in the RAIN

Posted on May 5, 2010 by Rob H

Or “behold the power of the unreliable”

In a previous post, I discussed the concept of a Redundant Array of Inexpensive Nodes (RAIN) as a way to create more reliable and scalable applications. Deploying a RAIN application is like being the House in Vegas – it’s about having enough size that the odds come out in your favor. Even if one player is on a roll, you can predict that nearly everyone else is paying your rent. Imagine what would happen if all the winning gamblers were in your casino! If you don’t want to go bankrupt when deploying a RAIN app, then ensure that the players spread out all over the Strip.

One of my core assumptions is that you’ll deploy a RAIN application on a cloud. This is a significant because we’re assuming that your nodes are

idle most of the time because your traffic loads are cyclic
unreliable because the cloud provider does not offer much SLA
divisible because renting ten 1/10^thsof a server costs roughly the same as a whole one
burstable because 1/10^th servers can sometimes consume that extra 9/10^th server

The burstable concept is a dramatic power multiplier that tips the whole RAIN equation heavily towards clouds.

Bursting means that under load, your 10 1/10^th servers (roughly 1 server in cost) could instantly expand to the power of 10 full servers! That reflects an order of magnitude spike in demand without any change in your application.

In the past, we’ve racked extra servers to handle this demand. That meant that we had a lot of extra capacity taking up rack space, clubbing innocent migratory electrons for their soft velvety fur, and committing over provisioning atrocities.

Today, multi-tenant clouds allow us to average out these bursts by playing the odds on when application bursts will occur. In this scenario, the infrastructure provider benefits from the fact that applications need to be over provisioned. The application author benefits because they can instantly tap more resources on demand. Now, that is what I would call a win-win synergy!

All this goodness depends on

standard patterns & practices that developers use to scale out RAIN applications
platform improvements in cloud infrastructure to enable smooth scale out
commercial models that fairly charge for multi-tenant over-subscription
workable security that understands the underlying need for co-mingling workloads

The growing dominance of cloud deployments looks different when you understand the multiplying interplay between multi-tenant clouds and cloud-ready RAIN applications.

Rethinking Storage

Posted on April 7, 2010 by Rob H

Or “UNthinking SANs”

Back in 2001, I was co-founder of a start-up building the first Internet virtualized cloud. Dual CPU 1U pizza box servers were brand new and we were ready to build out an 8 node, 64 VM cloud! It was going to be a dream – all that RAM and CPU just begging to be oversubscribed. It was enough to make Turing weep for joy.

Unfortunately, all those VMs needed lots and lots of storage.

Never fear, EMC was more than happy to quote us a lovely SAN with plenty of redundant FBAs and interconnected fabric switches. It was all so shiny and cool yet totally unscalable and obscenely expensive. Yes, unscalable because that nascent 8 node cloud was already at the port limit for the solution! Yes, expensive because that $50,000 hardware solution would have needed a $1,000,000 storage solution!

The funny part is that even after learning all that, we still wanted to buy the SAN. It was just that cool.

We never bought that SAN, but we did buy a very workable NAS device. Then it was my job to change (“pragmatic-ize”) our architecture so that our cloud management did not require expensive shiny objects.

Our ultimate solution used the NAS for master images that were accessed by many nodes. These requests were mainly reads and optimized. Writes were made to differencing disks kept on local disk and highly scalable. In systems, we were able to keep the masters local and save bandwidth. This same strategy could easily be applied in current “stateless” VM deployments.

Some of the SANless benefits are:

Less cost
Simplicity of networking and management
Nearer to linear scale out
Improved I/O throughput
Better fault tolerance (storage faults are isolated to individual nodes)

Of course, there are costs:

More spindles means more energy use (depending on drive selection and other factors)
Lack of centralized data management
Potentially wasted space because each system carries excess capacity
The need to synchronize data stored in multiple locations

These are real costs; however, I believe the data management problems are unsolved issues for SAN deployments too. Data proliferation is simply hidden inside of the VMs.

Today, I observe many different SAN focused architectures and cringe. These same solutions could be much simpler, more scalable and dramatically affordable with minimal (or even no) changes. If you’re serious about deploying a cloud based on commodity system then you need seriously need to re-evaluate your storage.

Dell goes to the Clouds (hardware & Joyent)

Posted on March 24, 2010 by Rob H

As a Dell employee, I’ve had the privilege of being on the front lines of Dell’s cloud strategy. Until today, I have not been able to post about the exciting offerings that we’ve been brewing.

Two related components have been occupying my days. The first is the new cloud optimized hardware and the second is the agreement to offer private clouds using Joyent’s infrastructure. Over the next few weeks, I’ll be exploring some of the implications of these technologies. I’ve already been exploring them in previous posts.

Cloud optimized hardware grew out of lesson learned in Dell’s custom mega-volume hardware business (that’s another story!). This hardware is built for applications and data centers that embrace scale out designs. These customers build applications that are so fault tolerant that they can focus on power, density, and cost optimizations instead of IT hardening. It’s a different way of looking at the data center because they see the applications and the hardware as a whole system.

To me, that system view is the soul of cloud computing.

The Dell-Joyent relationship is a departure from the expected. As a founder of Surgient, I’m no stranger to hypervisor private clouds; however, the Joyent takes a fundamentally different approach. Riding on top of OpenSolaris’ paravirtualization, this cloud solution virtually eliminates the overhead and complexity that seem to be the default for other virtualization solutions. I especially like Joyent’s application architectures and their persistent vision on how to build scale-out applications from the ground up.

To me, scale should be baked into the heart of cloud applications.

So when I look at Dell’s offings, I think we’ve captured the heart and soul of true cloud computing.

Rob’s rules for good APIs (with explanations!):

Posted on March 17, 2010 by Rob H

or “being a provider in an age of mass consumption.”

While blogging to abstraction, I kept thinking of ways to I should have avoided those awkward and painful APIs that I’ve written in the past. Short of a mouthful of bad tasting SOAP, here are my top API considerations.

Before you get started:

Take are to design the semantics of you API because APIs are abstraction that hide complexity from, simplify work of and encourage good behaviors in its consumers.
Think security and auditing first because it’s difficult to “see” all the options that an API will expose on a quick review. You don’t want deal with the mess when Venkman crosses the streams.
Write tests for your API because you want to find out that you broke it before your consumers do. Even better, also add simulators that pump data into your API so that you can run system tests.
Use versioned APIs even at 1.0 because you and your consumer must expect APIs to evolve.
Separate verbs (taking actions) from nouns (editing things) in your API because they have fundamentally different use, validation, and notification models. This also helps eliminate side-effects.
Do not anticipate consumers’ appetite for data because there is a cost for returning information and maintaining compatibility.

Now that you’re committed:

Offer the minimum options to service the consumer’s use-case because it’s easier to add than remove options.
Avoid exposing details to your consumer because it will lock-in your implementation and limit your flexibility.
Use natural keys over system assigned keys because users will have to make extra calls to figure out your assignments and it will lock your implementation.
Never break your API’s existing calls when adding options because you never really know who is using which API calls.
Object versioning (timestamping is enough!) reduces confusion because you can’t control how long a user will hang onto an object before they decide to send it back to you.
Fail loudly with obvious delight because you want to make it hard for bad data to get into your system and you also consumers to know what went wrong.
And, when failing loudly use HTTP error codes correctly because they really do mean something.
Finally, be text tolerant because strong typing makes your API harder to use and text is very robust and flexible.

Whew. Now for a REST.

Abstractions are great, except when they’re not

Posted on March 3, 2010 by Rob H

Or “please don’t make my life 100% easier, 80% is enough”

I had an interesting argument recently in a very crowded meeting – maybe we were all getting that purple meeting haze, but it started to take on all the makings of a holy war (so I knew it would make a good blog post).

We were discussing an API for interacting with a server cloud and the API was intentionally very abstracted. Specifically, you could manage a virtual server but you could not see which host was providing the resources. The vendor wanted to hide the raw resources from API consumers. This abstraction was good; it made the API simpler and allowed the provider to flexibility about how it implemented the backend. The API abstraction made the underlying system opaque.

So far it was all rainbows, unicorns and smiling yellow hatted yard gnomes.

Then I wanted to know if it was possible to relate information between the new API and the existing resource transparent API. Why would I want to do that? I was interested in the 5% case where we needed to get information about the specific resources that assigned. For example, when setting up redundant database replication, we want to make sure that they are not assigned to the same physical hosts.

More importantly, I do not want the vendor to clutter their new abstracted API with stuff to handle these odd ball use cases. Calling them 5% use-cases is deceptive: they are really in the hugely diverse bucket of use-cases outside of the 95% that are handled nicely the abstractions bucket. Trying to crow bar in these extra long tail use-cases will make the API unwieldy for the intended audience.

Someone else in the meeting disagreed with the premise of my question and wanted me to explain it. In answer, I used the tautology “Abstractions are useful, until they are not.”

The clearest example of this concept is the difference between Rails ActiveRecord and Hibernate. Both are excellent object-relational (OR) abstractions. The make the most general cases (select * from foo where ID = bar) quick and easy. But they are radically different at the edge of the abstraction. ActiveRecord expects that programmers will write directly to the database’s native SQL to handle the 5% exceptions. Hibernate added the complex and cumbersome HQL on top of their abstraction layer. HQL is nearly (in some cases, more) complex than the SQL language that it tries to abstract. For Hibernate, this is really an anti-abstraction that’s no longer useful.

Over stretching an abstraction encourages the wrong behaviors and leads to overly complex and difficult to maintain APIs. When you reach the edge of an abstraction, it’s healthy to peek under the covers. Chances are that you’re doing something wrong or something unique enough that you’ve outgrown the abstraction.

And that’s enough for now because blog posts are useful, until they are not.

Time vs. Materials: $1,000 printer power button

Posted on February 4, 2010 by Rob H

Or why I teach my kids to solder

I just spent four hours doing tech support over a $0.01 part on an $80 inkjet printer. According to my wife, those hours were a drop in the budget in a long line of comrades-in-geekdom who had been trying to get her office printer printing. All told, at least $1,000 worth of expert’s time was invested.

It really troubles me when the ratio of purchase cost to support cost exceeds 10x for a brand new device.

In this case, a stuck power button cover forced the printer into a cryptic QA test mode. It was obvious that the button was stuck, but not so obvious that that effectively crippled the printer. Ultimately, my 14 year old striped the printer down, removed the $0.01 button cover, accidentally stripped a cable, soldered it back together, and finally repaired the printer.

From a cost perspective, my wife’s office would have been exponentially smarter to dump the whole thing in to the trash and get a new one. Even the effort of returning it to the store was hardly worth the time lost dealing with the return.

This thinking really, really troubles me.

I have to wonder what it would cost our industry to create products that were field maintainable, easier to troubleshoot, and less likely to fail. The automotive industry seems to be ahead of us in some respects. They create products that a reliable, field maintainable, and conform to standards (given Toyota’s recent woes, do I need to reconsider this statement?). Unfortunately, they are slow to innovate and have become highly constrained by legislative oversight. Remember the old “If Microsoft made cars” joke?

For the high tech industry, I see systemic challenges driven by a number of market pressures:

Pace of innovation: our use of silicon is just graduating from crawling to baby steps. Products from the 90s look like stone tablets compared to 10’s offerings. This is not just lipstick, these innovations disrupt design processes making it expensive to maintain legacy systems.
Time to market: global competitive pressures to penetrate new markets give new customer acquisition design precedence.
Lack of standards: standards can’t keep up with innovation and market pressures. We’re growing to accept the consensus model for ad hoc standardization. Personally, I like this approach, but we’re still learning how to keep it fair.
System complexity: To make systems feature rich and cost effective, we make them tightly coupled. This is great at design time, but eliminates maintainability because it’s impossible to isolate and replace individual components.
Unequal wealth and labor rates: Our good fortune and high standard of living make it impractical for us to spend time repairing or upgrading. We save this labor by buying new products made in places where labor is cheap. These cheap goods often lack quality and the cycle repeats.
Inventory costs: Carrying low-demand, non-standard goods in inventory is expensive. I can a printer with thousands of resistors soldered onto a board for $89 while buying the same resistors alone would cost more than the whole printer. Can anyone afford to keep the parts needed for maintenance in stock?
Disposable resources: We deplete limited resources as if they were unlimited. Not even going to start on this rant…

Looking at these pressures makes the challenge appear overwhelming, but we need to find a way out of this trap.

That sounds like the subject for a future post!

Ready to Fail

Posted on January 23, 2010 by Rob H

Or How Monte Python taught me to program

Sometimes you learn the most from boring conference calls. In this case, I was listening to a deployment that was so painfully reference-example super-redundant by-the-book that I could have completed the presenter’s sentences. Except that he kept complaining about the cost. It turns out that our typical failure-proofed belt-and-suspenders infrastructure is really, really expensive.

Shouldn’t our applications be Monte Python’s Black Knight yelling “It’s just a flesh wound! Come back and fight!” Instead, we’ve grown to tolerate princess applications that throw a tantrum of over skim milk instead of organic soy in their mochaito.

Making an application failure-ready requires a mindset change. It means taking of our architecture space suit and donning our welding helmet.

Fragility is often born from complexity and complexity is the compounded interest from system design assumptions.

Let’s consider a transactional SQL database. I love relational databases. Really, I do. Just typing SELECT * FROM or LEFT OUTER JOIN gives me XKCD-like goose bumps. Unfortunately, they are as fragile as Cinderella’s glass slippers. The whole concept of relational databases requires a complex web of sophisticated data integrity we’ve been able to take for granted. The web requires intricate locking mechanisms that make data replication tricky. We could take it for granted because our operations people have built up super-complex triple-redundant infrastructure so that we did not have to consider what happens when the database can’t perform its magic.

What is the real cost for that magic?

I’m learning about CouchDB. It’s not a relational database, it a distributed JSON document warehouse with smart indexing. And compared some of the fine grained features of SQL, it’s an arc welder. The data in CouchDB is loosely structured (JSON!) and relationships are ad hoc. The system doesn’t care (let alone enforce) that if you’ve maintained referential integrity within the document – it just wants to make sure that the documents are stored, replicated, and indexed. The goodness here is that CouchDB allows you to distribute your data broadly so that it can be local and redundant. Even better, weak structure allows you to evolve your schema agilely (look for a future post on this topic).

If you’re cringing about lack referential integrity then get over it – every SQL backed application I ever wrote required RI double-checking anyway!

If you’re cringing about possible dirty reads or race conditions then get over it – every SQL backed application I ever wrote required collision protection too!

I’m not pitching CouchDB (or similar) is a SQL replacement. I’m holding it up as an example of a pragmatic approach to failure-ready design. I’m asking you to think about the hidden complexity and consequential fragility that you may blindly inherit.

So cut off my arms and legs – I can still spit on your shoes.

Rob Hirschfeld

On Computing, Containers, Cloud & Tech Culture

Category Archives: Architecture