Rethinking Storage

Posted on by
1

Or “UNthinking SANs”

Back in 2001, I was co-founder of a start-up building the first Internet virtualized cloud.  Dual CPU 1U pizza box servers were brand new and we were ready to build out an 8 node, 64 VM cloud!  It was going to be a dream – all that RAM and CPU just begging to be oversubscribed.  It was enough to make Turing weep for joy.

Unfortunately, all those VMs needed lots and lots of storage.

Never fear, EMC was more than happy to quote us a lovely SAN with plenty of redundant FBAs and interconnected fabric switches.  It was all so shiny and cool yet totally unscalable and obscenely expensive.   Yes, unscalable because that nascent 8 node cloud was already at the port limit for the solution!  Yes, expensive because that $50,000 hardware solution would have needed a $1,000,000 storage solution!

The funny part is that even after learning all that, we still wanted to buy the SAN.  It was just that cool.

We never bought that SAN, but we did buy a very workable NAS device.  Then it was my job to change (“pragmatic-ize”) our architecture so that our cloud management did not require expensive shiny objects.

Our ultimate solution used the NAS for master images that were accessed by many nodes.  These requests were mainly reads and optimized.  Writes were made to differencing disks kept on local disk and highly scalable.  In systems, we were able to keep the masters local and save bandwidth.  This same strategy could easily be applied in current “stateless” VM deployments.

Some of the SANless benefits are:

  • Less cost
  • Simplicity of networking and management
  • Nearer to linear scale out
  • Improved I/O throughput
  • Better fault tolerance (storage faults are isolated to individual nodes)

Of course, there are costs:

  • More spindles means more energy use (depending on drive selection and other factors)
  • Lack of centralized data management
  • Potentially wasted space because each system carries excess capacity
  • The need to synchronize data stored in multiple locations

These are real costs; however, I believe the data management problems are unsolved issues for SAN deployments too.  Data proliferation is simply hidden inside of the VMs.

Today, I observe many different SAN focused architectures and cringe.  These same solutions could be much simpler, more scalable and dramatically affordable with minimal (or even no) changes.  If you’re serious about deploying a cloud based on commodity system then you need seriously need to re-evaluate your storage.

McCrory: Zoo Tycoon = DC Management post

Posted on by
Reply

One of my past colleagues and fellow virtual server cloud inventor, Dave McCrory, posted an interesting comparison between Zoo Tycoon and the challenge of running a virtualized data center.

Way back in 2000, we talked about using this type of interface (or better a FPS!) to help manage data centers.  It still seems practical that the game dynamic of herding animals (VMs), visitors (users), attractions (infrastructure) ties very well to the daily activity of running a data center.

Looking that the landscape 10 years later, I think we’re still a long way from seeing that happen.  I don’t think it’s just a question of APIs – vendors need to look at the problem differently is we want to use this type of solution model.

Now, which start-up has the guts vision to give this approach a try?

Dell goes to the Clouds (hardware & Joyent)

Posted on by
1

As a Dell employee, I’ve had the privilege of being on the front lines of Dell’s cloud strategy.  Until today, I have not been able to post about the exciting offerings that we’ve been brewing.

Two related components have been occupying my days.  The first is the new cloud optimized hardware and the second is the agreement to offer private clouds using Joyent’s infrastructure. Over the next few weeks, I’ll be exploring some of the implications of these technologies.  I’ve already been exploring them in previous posts.

Cloud optimized hardware grew out of lesson learned in Dell’s custom mega-volume hardware business (that’s another story!).  This hardware is built for applications and data centers that embrace scale out designs.  These customers build applications that are so fault tolerant that they can focus on power, density, and cost optimizations instead of IT hardening.  It’s a different way of looking at the data center because they see the applications and the hardware as a whole system.

To me, that system view is the soul of cloud computing.

The Dell-Joyent relationship is a departure from the expected.  As a founder of Surgient, I’m no stranger to hypervisor private clouds; however, the Joyent takes a fundamentally different approach.  Riding on top of OpenSolaris’ paravirtualization, this cloud solution virtually eliminates the overhead and complexity that seem to be the default for other virtualization solutions.  I especially like Joyent’s application architectures and their persistent vision on how to build scale-out applications from the ground up.

To me, scale should be baked into the heart of cloud applications.

So when I look at Dell’s offings, I think we’ve captured the heart and soul of true cloud computing.

Cloud Application Life Cycle

Posted on by
Reply

Or “you learn by doing, and doing, and doing”

One of the most consistent comments I hear about cloud applications is that it fundamentally changes the way applications are written.  I’m not talking about the technologies, but the processes and infrastructure.

Since our underlying assumption of a cloud application is that node failure is expected then our development efforts need to build in that assumption before any code is written.  Consequently, cloud apps should be written directly on cloud infrastructure.

In old school development, I would have all the components for my application on my desktop.  That’s necessary for daily work, but does not give me a warm fuzzy for success in production.

Today’s scale production environments involve replicated data with synchronization lags, shared multi-writer memcache, load balancers, and mixed code versions.  There is no way that I can simulate that on my desktop!   There is no way I can fully anticipate how that will behave all together!

The traditional alternative is to wait.  Wait for QA to try and find bugs through trial and error.  Or (more likely) wait for users to discover the problem post deployment.

My alternative is to constantly deploy the application to a system that matches production.    As a bonus, I then attack the deployment with integration tests and simulators.

If you’re thinking that is too much effort then you are no thinking deeply enough.  This model forces developers to invest in install and deployment automation.  That means that you will be able to test earlier in the cycle.  It means you will be able to fix issues more quickly.  And that you’ll be able to ship more often.  It means that you can involve operations and networking specialists well before production.  You may even see more collaboration between your development, quality, and operations teams.   

Forget about that last one – if those teams actually worked together you might accidently ship product on time.  Gasp!

Rob’s rules for good APIs (with explanations!):

Posted on by
Reply

or “being a provider in an age of mass consumption.”

While blogging to abstraction, I kept thinking of ways to I should have avoided those awkward and painful APIs that I’ve written in the past.  Short of a mouthful of bad tasting SOAP, here are my top API considerations.

Before you get started:

  1. Take are to design the semantics of you API because APIs are abstraction that hide complexity from, simplify work of and encourage good behaviors in its consumers.
  2. Think security and auditing first because it’s difficult to “see” all the options that an API will expose on a quick review.  You don’t want deal with the mess when Venkman crosses the streams.
  3. Write tests for your API because you want to find out that you broke it before your consumers do.  Even better, also add simulators that pump data into your API so that you can run system tests.
  4. Use versioned APIs even at 1.0 because you and your consumer must expect APIs to evolve.
  5. Separate verbs (taking actions) from nouns (editing things) in your API because they have fundamentally different use, validation, and notification models.   This also helps eliminate side-effects.
  6. Do not anticipate consumers’ appetite for data because there is a cost for returning information and maintaining compatibility.

Now that you’re committed:

  1. Offer the minimum options to service the consumer’s use-case because it’s easier to add than remove options.
  2. Avoid exposing details to your consumer because it will lock-in your implementation and limit your flexibility.
  3. Use natural keys over system assigned keys because users will have to make extra calls to figure out your assignments and it will lock your implementation.
  4. Never break your API’s existing calls when adding options because you never really know who is using which API calls.
  5. Object versioning (timestamping is enough!) reduces confusion because you can’t control how long a user will hang onto an object before they decide to send it back to you.
  6. Fail loudly with obvious delight because you want to make it hard for bad data to get into your system and you also consumers to know what went wrong. 
  7. And, when failing loudly use HTTP error codes correctly because they really do mean something.
  8. Finally, be text tolerant because strong typing makes your API harder to use and text is very robust and flexible.

Whew.  Now for a REST.

Abstractions are great, except when they’re not

Posted on by
3

Or “please don’t make my life 100% easier, 80% is enough”

I had an interesting argument recently in a very crowded meeting – maybe we were all getting that purple meeting haze, but it started to take on all the makings of a holy war (so I knew it would make a good blog post).

We were discussing an API for interacting with a server cloud and the API was intentionally very abstracted.  Specifically, you could manage a virtual server but you could not see which host was providing the resources.  The vendor wanted to hide the raw resources from API consumers.  This abstraction was good; it made the API simpler and allowed the provider to flexibility about how it implemented the backend.  The API abstraction made the underlying system opaque.

So far it was all rainbows, unicorns and smiling yellow hatted yard gnomes.

Then I wanted to know if it was possible to relate information between the new API and the existing resource transparent API.  Why would I want to do that?  I was interested in the 5% case where we needed to get information about the specific resources that assigned.  For example, when setting up redundant database replication, we want to make sure that they are not assigned to the same physical hosts.

More importantly, I do not want the vendor to clutter their new abstracted API with stuff to handle these odd ball use cases.  Calling them 5% use-cases is deceptive: they are really in the hugely diverse bucket of use-cases outside of the 95% that are handled nicely the abstractions bucket.  Trying to crow bar in these extra long tail use-cases will make the API unwieldy for the intended audience. 

Someone else in the meeting disagreed with the premise of my question and wanted me to explain it.  In answer, I used the tautology “Abstractions are useful, until they are not.”

The clearest example of this concept is the difference between Rails ActiveRecord and Hibernate.  Both are excellent object-relational (OR) abstractions.   The make the most general cases (select * from foo where ID = bar) quick and easy.   But they are radically different at the edge of the abstraction.  ActiveRecord expects that programmers will write directly to the database’s native SQL to handle the 5% exceptions.  Hibernate added the complex and cumbersome HQL on top of their abstraction layer.  HQL is nearly (in some cases, more) complex than the SQL language that it tries to abstract.  For Hibernate, this is really an anti-abstraction that’s no longer useful.

 Over stretching an abstraction encourages the wrong behaviors and leads to overly complex and difficult to maintain APIs.  When you reach the edge of an abstraction, it’s healthy to peek under the covers.  Chances are that you’re doing something wrong or something unique enough that you’ve outgrown the abstraction.

And that’s enough for now because blog posts are useful, until they are not.

IEEE “Pragmatic Clouds” Presentation (2/24/10)

Posted on by
Reply

Tonight I presentated “Pragmatic Clouds” to the IEEE Central Texas Consultants Network Meeting.

The abstract was, “Cloud computing seems to mean everything! We’ll talk in specifics about how application development and delivery models are changing based on new “cloud” commercial models. We’ll also explore how the plumbing behind the curtains is changing to reflect these new models.”

In the presentation, I covered drivers for cloud business models and how it impacts creating applications for clouds. I also described how to write a future-proof application that can work for IaaS clouds today and PaaS clouds tomorrow.

WhatTheBus gets its move on: live maps arrive

Posted on by
Reply

The “livemap” tag for WhatTheBus completes the work that I seeded in the last commit.

In the last update, we had finished a page that used jQuery AJAX updates to get the latest bus location from the cache.  Using the simulator to provide updates, the AJAX updates shows that we could get location updates as latitude and longitudes.  By watching the dev web server, I also saw that these requests were super light (0ms DB, 0ms view).

In this update, I took the same basic code and added Google maps (v3) interaction.  Using Google and jQuery together maps is refreshingly easy.

The first step was to render the map when the page loads.  This required adding the Google javascript library and pointing the page’s onload event  to initMap.  The initMap function creates a new map object centered on the bus’ location, creates a bus marker at the center, and then replaces the page’s live_map div with the map.

<script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=true&amp;key=<%= MAP_KEY || "not_set_in_#{RAILS_ENV}_config" %>"> </script>
<script type="text/javascript">
var map;
var busMarker;
var xref = '<%= @bus.xref %>';
var name = '<%= @bus.name %>';
var tstamp = new Date();
function initMap() {
  var centerCoord = new google.maps.LatLng(<%= @pos %>);
  var mapOptions = {
    zoom: 16,
    center: centerCoord,
    mapTypeId: google.maps.MapTypeId.ROADMAP
  };
  map = new google.maps.Map(document.getElementById("live_map"), mapOptions);
  busMarker = new google.maps.Marker({
    position: centerCoord,
    map: map,
    title: name,
    icon: "/images/bus.png"
  });
  window.setInterval('updateMap();', <%= MAP_UPDATE_SECS %>000);
}

The second step was to add the AJAX call on a timer.  After adding the update function timer registration, the updates simply extended the existing AJAX request.  This existing request already had the bus’ position so the work centered on interacting with the map.

To keep things friendly, we turn the last marker into a bullet point and change the bus name to the last time.  Then we create a new marker based on that latest position and center the map on that position too.  To prevent reduce server load for non-reporting buses, we move off the map page if there is no position data.

function updateMap() {
  busMarker.setIcon("/images/track.png");
  busMarker.setTitle(tstamp.getHours() + ":" + tstamp.getMinutes() + ":" + tstamp.getSeconds());
  tstamp = new Date();
  // get the data for the map
  jQuery.getJSON("/bus/index/"+ xref +".json?cache", {}, function(data){
    if (data.length==0) window.location.href("/bus/index/<%= @bus.xref %>");
      var newCoord = new google.maps.LatLng(parseFloat(data.buses[xref].lat), parseFloat(data.buses[xref].lng));
      busMarker = new google.maps.Marker({
        position: newCoord,
        map: map,
        title: name,
        icon: "/images/bus.png"
      });
      map.setCenter(newCoord);
  });
}

After a few tweaks to add navigation between the existing pages, the first use-case of WhatTheBus is in the bag!

The next step is to setup the code online and get a district to send updates!

Time vs. Materials: $1,000 printer power button

Posted on by
Reply

Or why I teach my kids to solder

I just spent four hours doing tech support over a $0.01 part on an $80 inkjet printer.  According to my wife, those hours were a drop in the budget in a long line of comrades-in-geekdom who had been trying to get her office printer printing.  All told, at least $1,000 worth of expert’s time was invested.

It really troubles me when the ratio of purchase cost to support cost exceeds 10x for a brand new device.

In this case, a stuck power button cover forced the printer into a cryptic QA test mode.  It was obvious that the button was stuck, but not so obvious that that effectively crippled the printer.   Ultimately, my 14 year old striped the printer down, removed the $0.01 button cover, accidentally stripped a cable, soldered it back together, and finally repaired the printer.

From a cost perspective, my wife’s office would have been exponentially smarter to dump the whole thing in to the trash and get a new one.   Even the effort of returning it to the store was hardly worth the time lost dealing with the return.

This thinking really, really troubles me.

I have to wonder what it would cost our industry to create products that were field maintainable, easier to troubleshoot, and less likely to fail.  The automotive industry seems to be ahead of us in some respects.  They create products that a reliable, field maintainable, and conform to standards (given Toyota’s recent woes, do I need to reconsider this statement?).  Unfortunately, they are slow to innovate and have become highly constrained by legislative oversight.  Remember the old “If Microsoft made cars” joke?

For the high tech industry, I see systemic challenges driven by a number of market pressures:

  1. Pace of innovation: our use of silicon is just graduating from crawling to baby steps.  Products from the 90s look like stone tablets compared to 10’s offerings.   This is not just lipstick, these innovations disrupt design processes making it expensive to maintain legacy systems.
  2. Time to market: global competitive pressures to penetrate new markets give new customer acquisition design precedence.
  3. Lack of standards: standards can’t keep up with innovation and market pressures.  We’re growing to accept the consensus model for ad hoc standardization.  Personally, I like this approach, but we’re still learning how to keep it fair.
  4. System complexity: To make systems feature rich and cost effective, we make them tightly coupled.  This is great at design time, but eliminates maintainability because it’s impossible to isolate and replace individual components.
  5. Unequal wealth and labor rates:  Our good fortune and high standard of living make it impractical for us to spend time repairing or upgrading.  We save this labor by buying new products made in places where labor is cheap.  These cheap goods often lack quality and the cycle repeats.
  6. Inventory costs: Carrying low-demand, non-standard goods in inventory is expensive.   I can a printer with thousands of resistors soldered onto a board for $89 while buying the same resistors alone would cost more than the whole printer.  Can anyone afford to keep the parts needed for maintenance in stock?
  7. Disposable resources: We deplete limited resources as if they were unlimited.  Not even going to start on this rant…

Looking at these pressures makes the challenge appear overwhelming, but we need to find a way out of this trap.

That sounds like the subject for a future post!

WhatTheBus Map seeds planted, looks like Cucumbers

Posted on by
Reply

Today’s WhatTheBus progress (see tag “MapSeeds”) created the foundation for live maps. These maps will only show a single bus at a time because our initial use case is that a parent wants to monitor their child’s bus.

To accomplish this use case, we need to provide a list of all the buses in the system (grouped by district) and then allow users to select a bus and see the map.

This foundation started with a series of Cucumber tests that verified the navigation structure and the base map page.  These pages used the existing page tests (yawn).

In fact, most of this work is pretty dull Rails web navigation stuff.  The interesting parts were:

  1. DRY the bus location from cache by moving it into the model
  2. Add jRails so that we can use jQuery goodness
  3. Optimization: added index for Bus xref
  4. Optimization: added ?cache flag for the bus json requests to bypass database calls on location only requests

In our next pass, we’ll add support for Google Maps (v3).  Until then, we just have some simple script that pulls the current bus position using our existing bus JSON request.  I am using the simulator (“rake sim:move”) to verify this; unfortunately, Cucumber does have native AJAX support.

Here’s the <script>


<script type="text/javascript">
  var xref = '<%= @bus.xref %>';
  var name = '<%= @bus.name %>';

  function initMap() {
    window.setInterval('updateMap();', <%= MAP_UPDATE_SECS %>000);
    updateMap();
  }

  function updateMap() {
    jQuery("#ll").text("?,?");

    // get the data for the map
    jQuery.getJSON("/bus/index/"+ xref +".json?cache", {}, function(data){
    jQuery("#ll").text(data.buses[xref].lat + ',' + data.buses[xref].lng);
    });
  }
</script>