My OpenStack 2016 Analysis: Continue Core, Stop Confusing Ecosystem, Change Hybrid Approach

Posted on December 30, 2015 by Rob H

Note: I’ve served on the OpenStack Foundation board since its formation. There I’ve led the “define the core” DefCore efforts. I’m on the 2016 ballot for another term.

I love using end-of-year posts to reflect (2015, I got 6 of 7!) and try to set direction (OpenStack needed to prioritize). This year, I wanted to use a simple “Continue, Stop, Change” format that I’ve used for employee reviews in the past. These three items reflect how I think OpenStack needs to respond to the industry in 206.

Continue: Focus on Core

OpenStack adoption continues around the legacy projects that traditionally define it for most users. A lot of work and focus is needed around those projects including better representation of user, operator and product interests.

Towards that end, we’ve made amazing progress on DefCore implementation and I’m excited about the discussions that it’s been generating. It’s driving pragmatic decisions about what is required (running a vm?) and how to verify compliance. It’s also driving conceptual thinking around OpenStack principles and ecosystem priorities.

DefCore’s focus on using community tests to define OpenStack creates a very concrete and defensible standard. Ultimately, it comes back to users and operators demanding compliance for the work to remain meaningful.

Overall, To focus on core function, OpenStack needs to empower new groups within the community. Expanding the role of the Product Group, Operators, and User Committee are key to giving a voice to these constituents.

OpenStack core must transition into a consistent platform or it risks becoming irrelevant.

Stop: Confusing The Ecosystem

I’m concerned about the “big tent” governance change puts OpenStack into conflict with both community vendors and the larger cloud market. I believe we’re creating an echo chamber of OpenStack on OpenStack focus that forces adjacent efforts (like software defined network, storage and container orchestration) to be either inside or outside the community circle. While that artificially grows the apparent contributor base, it creates artificial walls between OpenStack and the dominate cloud platforms.

Let me illustrate using my own company, RackN. We create cross-platform devops orchestration based on an open source project, Digital Rebar. We consider ourselves to be part of the OpenStack community and have supported deploying the core. We also provision bare metal and deploy Kubernetes, Docker Swarm and Cloud Foundry. That has apparent conflicts with big tent Ironic and Magnum projects. Does that make RackN competitive with OpenStack or not?

It hurts OpenStack when competitive alignment is unclear because vendors, users and operators are uncertain about where to make investments. In the end, users will choose simpler alternatives.

I believe the Board needs to define the OpenStack ecosystem strategy in a clear and actionable way. If re-elected, that will be my Board priority for 2016.

Change: Hybrid Approach

My top 2016 prediction (post coming) is that we accept “hybrid IT as the new normal.” That means that we stop driving towards an IT mono-culture and start working towards tools that embrace heterogeneity. Along those lines, OpenStack needs to evaluate our relative position and strengths in a hybrid cloud landscape.

Interoperability between OpenStack implementations is important because it reduces friction; however, we need to expand our thinking to ensure interoperability with other platforms. That does not mean simply cloning the AWS APIs! It means that we need to consider users and operator needs against a spectrum of private and public infrastructures.

A broader hybrid approach also suggests that duplicating cloud-locked adjacent services (e.g. Cloud Formation vs. Heat) does not address user needs.

I am advocating that OpenStack encourage a cloud-neutral ecosystem, outside of the OpenStack tent, that work across a wide range of platforms. That leads to user choice and creates a truly open platform.

And, of course, more Community Discussion!

I want to thank the many people who participated in a heated twitter discussion in advance of this post. There are many great ideas and counter-points covered in that lengthy dialog.

Do you have an opinion about what to OpenStack should stop, accelerate or change? I’d love to hear it!

10 ways to make OpenStack more Start-up Friendly [even more critical in wake of recent consolidation]

Posted on June 3, 2015 by Rob H

The Josh McKenty comment that OpenStack is “aggressively anti-startup” for Business Insider got me thinking and today’s news about IBM & Cisco acquiring startups Blue Box & Piston made me decide to early release this post.

I think there’s a general confusion about start-ups in OpenStack. Many of the early (and now acquired) start-ups were selling OpenStack the platform. Since OpenStack is community infrastructure, that’s a really hard place to differentiate. Unfortunately, there’s no material install base (yet) to create an ecosystem of start-ups on top of OpenStack.

The real question is not how to make OpenStack start-up friendly, but how to create a thriving system around OpenStack like Amazon and VMware have created.

That said, here’s my list of ten ways that OpenStack could be more start-up friendly:

Accept companies will have some closed tech – Many investors believe that companies need proprietary IP. An “open all things” company will have more trouble with investors.
Stop scoring commits as community currency – Small companies don’t show up in the OpenStack committer economy because they are 1) small and 2) working on their product upstream ahead of OpenStack upstream code.
Have start-up travel assistance – OpenStack demands a lot of travel and start-ups don’t have the funds to chase the world-wide summits and mid-cycles.
Embrace open projects outside of OpenStack governance – Not all companies want or need that type of governance for their start-up code base. That does not make them less valuable, it just makes them not ready yet.
Stop anointing ecosystem projects as OpenStack projects – Projects that are allowed into OpenStack get to grab to a megaphone even if they have minimal feature sets.
Be language neutral – Python is not the only language and start-ups need to make practical choices based on their objectives, staff and architecture.
Have a stable base – start-ups don’t have time to troubleshoot both their own product and OpenStack. Without core stability, it’s risky to add OpenStack as a product requirement.
Focus on interoperability – Start-ups don’t have time evangelize OpenStack. They need OpenStack to have large base of public and private installs because that creates an addressable market.
Limit big companies from making big pre-announcements – Start-ups primary advantage is being a first/fast mover. When OpenStack members make announcements of intention (generally without substance) it damages the market for start-ups. Normally corporate announcements are just noise but they are given credibility when they appear to come from the community.
Reduce the contribution tax and patch backlog – Start-ups must seek the path of least friction. If needed OpenStack code changes require a lot of work and time then they are unlikely to look for less expensive alternatives.

While I believe these items would help start-ups, they would have negative consequences for the large corporate contributors who have fashioned OpenStack into the type of project that supports their needs.

I’d love to what items you think I’ve overlooked or incorrectly added.

Hidden costs of Cloud? No surprises, it’s still about complexity = people cost

Posted on May 4, 2015 by Rob H

Last week, Forbes and ZDnet posted articles discussing the cost of various cloud (451 source material behind wall) full of dollar per hour costs analysis. Their analysis talks about private infrastructure being an order of magnitude cheaper (yes, cheaper) to own than public cloud; however, the open source price advantages offered by OpenStack are swallowed by added cost of finding skilled operators and its lack of maturity.

At the end of the day, operational concerns are the differential factor.

The Magic 8 Cube

These articles get tied down into trying to normalize clouds to $/vm/hour analysis and buried the lead that the operational decisions about what contributes to cloud operational costs. I explored this a while back in my “magic 8 cube” series about six added management variations between public and private clouds.

In most cases, operations decisions is not just about cost – they factor in flexibility, stability and organizational readiness. From that perspective, the additional costs of public clouds and well-known stacks (VMware) are easily justified for smaller operations. Using alternatives means paying higher salaries and finding talent that requires larger scale to justify.

Operational complexity is a material cost that strongly detracts from new platforms (yes, OpenStack – we need to address this!)

Unfortunately, it’s hard for people building platforms to perceive the complexity experienced by people outside their community. We need to make sure that stability and operability are top line features because complexity adds a very real cost because it comes directly back to cost of operation.

In my thinking, the winners will be solutions that reduce BOTH cost and complexity. I’ve talked about that in the past and see the trend accelerating as more and more companies invest in ops automation.

Research showing that Short Lived Servers (“mayflies”) create efficiency at scale [DATA REQUESTED]

Posted on January 8, 2015 by Rob H

Last summer, Josh McKenty and I extended the puppies and cattle metaphor to limited life cattle we called “mayflies.” It was an attempt to help drive the cattle mindset (I think of it as social engineering, or maybe PsychOps) by forcing churn. I’ve come to think of it a step in between cattle and chaos monkeys (see Adrian Cockcroft).

While our thoughts were on mainly ops patterns, I’ve heard that there could be a real operational benefit from encouraging this behavior. The increased turn over in the environment improves scheduler optimization, planned load drains and coping with platform/environment migration.

Now we have a chance to quantify this benefit: a college student (disclosure: he’s my son) has created a data center emulation to see if Mayflies help with utilization. His model appears to work.

Now, he needs some real world data, here’s his request for assistance [note: he needs data by 1/20 to be included in this term]:

Hello!

I am Alexander Hirschfeld, a freshman at Rose-Hulman Institute of Technology. I am working on an independent study about Mayflies, a new idea in virtual machine management in cloud computing. Part of this management is load balancing and resource allocation for virtual machines across a collection of servers. The emulation that I am working on needs a realistic set of data to be the most accurate when modeling the results of using the methods outlined by the theory of mayflies.

Mayflies are an extension of the puppies verses cattle approach to machines, they are the extreme version of cattle as they have a known limited lifespan, such as 7 days. This requires the users of the cloud to build inherently more automated and fault-resistant applications. If you could send me a collection of the requests for new virtual machines(per standard unit of time and their requested specs/size), as well as an average lifetime for the virtual machines (or a graph or list of designated/estimated life times), and a basic summary of the collection of servers running the virtual machines(number, ram, cores), I would be better able to understand how Mayflies can affect a cloud.

Thanks,
Alexander Hirschfeld, twitter: @d-qoi

Needless to say, I’m really excited about the progress on demonstrating some the impact of this practice and am looking forward to posting about his results in the near future.

If you post in the comments, I will make sure you are connected to Alex.

10 pounds of OpenStack cloud in a 5 pound bag? Do we need a bigger bag?

Posted on December 31, 2014 by Rob H

Yesterday, I posted about cloud distruptors that are pushing the boundaries of cloud. The same forces pull at OpenStack where we are working to balance between including all aspects of running workloads and focusing on a stable foundation.

Note: I am seeking re-election to the 2015 OpenStack Board. Voting starts 1/12.

For weeks, I’ve been reading and listening to people inside and outside the community. There is considerable angst about the direction of OpenStack. We need to be honest and positive about challenges without simply throwing stones in our hall of mirrors.

Closing 2014, OpenStack has gotten very big, very fast. We’ve exploded scope, contributions and commercial participants. Unfortunately, our process infrastructure (especially the governance by-laws) simply have not kept pace. It’s not a matter of scaling processes we’ve got; many of the challenges created by growth require new approaches and thinking (Thierry’s post).

In 2015, we’re trying to put 10 pounds of OpenStack in a 5 pound bag. That means we have to either a) shed 5 pounds or b) get a bigger bag. In classic OpenStack style, we’re sort of doing both: identifying a foundational base while expanding to allow more subprojects.

To my ear, most users, operators and business people would like to see the focus being on the getting the integrated release scope solid. So, in spirit of finding 5 pounds to shave, I’ve got five “shovel ready” items that should help:

Prioritizing stability as our #1 feature. Accomplishing this will require across the broad alignment of the vendor’s product managers to hold back on their individual priorities in favor of community. We’ve started this effort but it’s going to take time to create the collaboration needed.
Sending a clear signal about the required baseline for OpenStack. That’s the purpose of DefCore and should be felt as we work on the Icehouse and Juno definitions.
Alignment of the Board DefCore project with Technical Committee’s Levels/Big Tent initiative. By design, these efforts interconnect. We need to make sure the work is coordinated so that we send a clearly aligned message to the technical, operator, vendor and user communities.
Accelerate changes from single node gate to something that’s either a) more services focused or b) multi-node. OpenStack’s scale of community development requires automation to validate the new contributions do not harm the existing code base (the gate). The current single-node gate does not reflect the multi-node environments that users target with the code. While it’s technically challenging to address this mismatch, it’s also essential so we ensure that we’re able to validate multi-node features.
Continue to reduce drama in the open source processes. OpenStack is infrastructure software that should enable an exciting and dynamic next generation of IT. I hear people talk about CloudStack as “it’s not as exciting or active a community but their stuff just works.” That’s what enterprises and operators want. Drama is great for grabbing headings but not so great for building solid infrastructure.

What is the downside to OpenStack if we cannot accomplish these changes? Forks.

I already see a clear pattern where vendors are creating their own distros (which are basically shallow forks) to preserve their own delivery cycle. OpenStack’s success is tied to its utility for the customers of vendors who fund the contributors. When the cost of being part of the community outweighs the value, those shallow forks may become true independent products.

In the case of potential forks, they allow vendors to create their own bag and pick how many pounds of cloud they want to carry. It’s our job as a community in 2015 to make sure that we’ve reduced that temptation.

1/9/15 Note: Here’s the original analogy image used for this post

2015, the year cloud died. Meet the seven riders of the cloudocalypse

Posted on December 30, 2014 by Rob H

After writing pages of notes about the impact of Docker, microservice architectures, mainstreaming of Ops Automation, software defined networking, exponential data growth and the explosion of alternative hardware architecture, I realized that it all boils down to the death of cloud as we know it.

OK, we’re not killing cloud per se this year. It’s more that we’ve put 10 pounds of cloud into a 5 pound bag so it’s just not working in 2015 to call it cloud.

Cloud was happily misunderstood back in 2012 as virtualized infrastructure wrapped in an API beside some platform services (like object storage).

That illusion will be shattered in 2015 as we fully digest the extent of the beautiful and complex mess that we’ve created in the search for better scale economics and faster delivery pipelines. 2015 is going to cause a lot of indigestion for CIOs, analysts and wandering technology executives. No one can pick the winners with Decisive Leadership™ alone because there are simply too many possible right ways to solve problems.

Here’s my list of the seven cloud disrupting technologies and frameworks that will gain even greater momentum in 2015:

Docker – I think that Docker is the face of a larger disruption around containers and packaging. I’m sure Docker is not the thing alone. There are a fleet of related technologies and Docker replacements; however, there’s no doubt that it’s leading a timely rethinking of application life-cycle delivery.
New languages and frameworks – it’s not just the rapid maturity of Node.js and Go, but the frameworks and services that we’re building (like Cloud Foundry or Apache Spark) that change the way we use traditional languages.
Microservice architectures – this is more than containers, it’s really Functional Programming for Ops (aka FuncOps) that’s a new generation of service oriented architecture that is being empowered by container orchestration systems (like Brooklyn or Fleet). Using microservices well seems to redefine how we use traditional cloud.
Mainstreaming of Ops Automation – We’re past “if DevOps” and into the how. Ops automation, not cloud, is the real puppies vs cattle battle ground. As IT creates automation to better use clouds, we create application portability that makes cloud disappear. This freedom translates into new choices (like PaaS, containers or hardware) for operators.
Software defined networking – SDN means different things but the impacts are all the same: we are automating networking and integrating it into our deployments. The days of networking and compute silos are ending and that’s going to change how we think about cloud and the supporting infrastructure.
Exponential data growth – you cannot build applications or infrastructure without considering how your storage needs will grow as we absorb more data streams and internet of things sources.
Explosion of alternative hardware architecture – In 2010, infrastructure was basically pizza box or blade from a handful of vendors. Today, I’m seeing a rising tide of alternatives architectures including ARM, Converged and Storage focused from an increasing cadre of sources including vendors sharing open designs (OCP). With improved automation, these new “non-cloud” options become part of the dynamic infrastructure spectrum.

Today these seven items create complexity and confusion as we work to balance the new concepts and technologies. I can see a path forward that redefines IT to be both more flexible and dynamic while also being stable and performing.

Want more 2015 predictions? Here’s my OpenStack EOY post about limiting/expanding the project scope.

Networking in Cloud Environments, SDN, NFV, and why it matters [part 2 of 2]

Posted on May 5, 2014 by Rob H

Scott Jensen is an Engineering Director and colleague of mine from Dell with deep networking and operations experience. He had first hand experience deploying OpenStack and Hadoop and has a critical role in defining Dell’s Reference Architectures in those areas. When I saw this writeup about cloud networking (first post), I asked if it would be OK to post it here and share it with you.

GUEST POST 2 OF 2 BY SCOTT JENSEN:

So what is different about Cloud and how does it impact on the network

In a traditional data center this was not all that difficult (relatively). You knew what was going to running on what system (physically) and could plan your infrastructure accordingly. The majority of the traffic moved in a North/South direction. Or basically from outside the infrastructure (the internet for example) to inside and then responded back out. You knew that if you had to design a communication channel from an application server to a database server this could be isolated from the other traffic as they did not usually reside on the same system.

Virtualization made this more difficult. In this model you are sharing systems resources for different applications. From the networks point of view there are a large number of systems available behind a couple of links. Live Migration puts another wrinkle in the design as you now have to deal with a specific system moving from one physical server to another. Network Virtualization helps out a lot with this. With this you can now move virtual ports from one physical server to another to ensure that when one virtual machine moves from a physical server to another that the network is still available. In many cases you managed these virtual networks the same as you managed your physical network. As a matter of fact they were designed to emulate the physical as much as possible. The virtual machines still looked a lot like the physical ones they replaced and can be treated in vary much the same way from a traffic flow perspective. The traffic still is primarily a North/South pattern.

Cloud, however, is a different ball of wax. Think about the charistics of the Cattle described above. A cloud application is smaller and purpose built. The majority of its traffic is between VMs as different tiers which were traditionally on the same system or in the same VM are now spread across multiple VMs. Therefore its traffic patterns are primarily East/West. You cannot forget that there is a North/South pattern the same as what was in the other models which is typically user interaction. It is stateless so that many copies of itself can run in tandem allowing it to elastically scale up and down based on need and as such they are appearing and disappearing from the network. As these VMs are spawned on the system they may be right next to each other or on different servers or potentially in different Data Centers. But it gets even better. scj-net2

Cloud architectures are typically multi-tenant. This means that multiple customers will utilize this infrastructure and need to be isolated from each other. And of course Clouds are self-service. Users/developers can design, build and deploy whenever they want. Including designing the network interconnects that their applications need to function. All of this will cause overlapping IP address domains, multiple virtual networks both L2 and L3, requirements for dynamically configuring QOS, Load Balancers and Firewalls. Lastly in our list of headaches is not the least. Cloud systems tend to breed like rabbits or multiply like coat hangers in the closet. There are more and more systems as 10 servers become 40 which becomes 100 then 1000 and so on.

So what is a poor Network Engineer to do?

First get a handle on what this Cloud thing is supposed to be for. If you are one of the lucky ones who can dictate the use of the infrastructure then rock on! Unfortunately, that does not seem to be the way it goes for many. In the case where you just cannot predict how the infrastructure will be used I am reminded of the phrase “there is not replacement for displacement”. Fast links, non-blocking switches, Network Fabrics are all necessary for the physical network but will not get you there. Sense as a network administrator you cannot predict the traffic patterns who can? Well the developer and the application itself. This is what SDN is all about. It allows a programmatic interface to what is called an overlay network. A series of tunnels/flows which can build virtual networks on top of the physical network giving that pesky application what it was looking for. In some cases you may want to make changes to the physical infrastructure. For example change the configuration of the Firewall or Load Balancer or other network equipment. SDN vendors are creating plug-ins that can make those types of configurations. But if this is not good enough for you there is NFV. The basic idea here is that why have specialized hardware for your core network infrastructure when we can run them virtualized as well? Let’s run those in VM’s as well, hook them into the virtual network and SDN to configure them and we now can virtualize the routers, load balancers, firewalls and switches. These technologies are in very much a state of flux right now but they are promising none the less. Now if we could just virtualize the monitoring and troubleshooting of these environments I’d be happy.

Networking in Cloud Environments, SDN, NFV, and why it matters [part 1 of 2]

Posted on May 1, 2014 by Rob H

Guest Post 1 of 2 by Scott Jensen:

Having a basis in enterprise data center networking, Cloud computing I have many conversations with customers implementing a cloud infrastructure. Their design the networking infrastructure can and should be different from a classic network configuration and many do not understand why. Either due to a lack of knowledge in networking or due to a lack of understanding as to why cloud computing is different from virtualization. Once you have an understanding of both of these areas you can begin to see why emerging technologies such as SDN (Software Defined Networking) and NFV (Network Function Virtualization) begin to address some of the issues that Cloud Computing can cause with your network.

Networking is all about traffic flows. In order to properly design your infrastructure you need to understand where traffic is originating, where it is going and how much traffic will be following a specific route and at what times.

There are many differences between Cloud Computing and virtualization. In many cases people I will talk to think of Cloud as virtualization in a different environment. Of course this will work just fine however it does not take advantage of the goodness that a Cloud infrastructure can bring. Some of the major differences between Virtualization and Cloud Computing have profound effects on how the network is utilized. This all has to do with the application. That is really what it is all about anyway. Rob Hirschfeld has a great post on the difference between Pets and Cattle which describes this well.

Pets and Cattle as a workload evolution

In typical virtualized infrastructures, the applications have a fairly common pattern. Many people describe these as Pets and are managed largely the same as a physical system. They have a name, they are one of a kind, they are cared for, and when the die it can be traumatic (I know I have been there).

They run on large stateful VMs
They have a lifecycle which is typically very long such as years
The applications themselves are not designed to tolerate failures. Other technologies are brought in to ensure uptime.
The application is scaled up when demands increase. This is done by adding more memory or CPU to the VM.

Cloud applications are different. Some people describe them as cattle and they are treated like cattle in many ways. They do not necessarily have a name and if one dies it is sad but not a really big deal. We should probably figure out what killed it but life goes on.

They run on smaller stateless VMs
They have a lifecycle measured in hours or months. Sometimes even less than an hour.
The application is designed to expect failures
The application scales out by increasing the number of instances which is running when the demand increases.

In his follow-up post next week, Scott discusses how this impacts the network and how SDN and NFV promises to help.

Success Factors of Operating Open Source Infrastructure [Series Intro]

Posted on April 29, 2014 by Rob H

Building a best practices platform is essential to helping companies share operations knowledge. In the fast-moving world of open source software, sharing documentation about what to do is not sufficient. We must share the how to do it also because the operations process is tightly coupled to achieving ongoing success.

Further, since change is constant, we need to change our definition of “stability” to reflect a much more iterative and fluid environment.

Baseline testing is an essential part of this platform. It enables customers to ensure not only fast time to value, but the tests are consistently conforming with industry best practices, even as the system is upgraded and migrates towards a continuous deployment infrastructure.

The details are too long for a single post so I’m going to explore this as three distinct topics over the next two weeks.

Reference Deployments talks about needed an automated way to repeat configuration between sites.
Ops Validation using Development Tests talks about having a way to verify that everyone uses a common reference platform
Shared Open Operatons / DevOps (pending) talks about putting reference deployment and common validation together to create a true open operations practice.

OpenStack, Hadoop, Ceph, Docker and other open source projects are changing the landscape for information technology. Customers seeking to become successful with these evolving platforms must look beyond the software bits, and consider both the culture and operations. The culture is critical because interacting with the open source projects community (directly or through a proxy) can help ensure success using the software. Operations are critical because open source projects expect the community to help find and resolve issues. This results in more robust and capable products. Consequently, users of open source software must operate in a more fluid environment.

My team at Dell saw this need as we navigated the early days of OpenStack. The Crowbar project started because we saw that the community needed a platform that could adapt and evolve with the open source projects that our advanced customers were implementing. Our ability to deliver an open operations platform enables the community to collaborate, and to skip over routine details to refocus on shared best practices.

My recent focus on the OpenStack DefCore work reinforces these original goals. Using tests to help provide a common baseline is a concrete, open and referenceable way to promote interoperability. I hope that this in turn drives a dialog around best practices and shared operations because those help mature the community.