Exploring the Edge Series: “Edge is NOT just Mini-Cloud”

Posted on September 20, 2017 by Rob H

While the RackN team and I have been heads down radically simplifying physical data center automation, I’ve still been tracking some key cloud infrastructure areas. One of the more interesting ones to me is Edge Infrastructure.

This once obscure topic has come front and center based on coming computing stress from home video, retail machine and distributed IoT. It’s clear that these are not solved from centralized data centers.

While I’m posting primarily on the RackN.com blog, I like to take time to bring critical items back to my personal blog as a collection. WARNIING: Some of these statements run counter to other industry. Please let me know what you think!

Don’t want to read? Here’s a summary podcast.

Post 1: OpenStack On Edge? 4 Ways Edge Is Distinct From Cloud

By far the largest issue of the Edge discussion was actually agreeing about what “edge” meant. It seemed as if every session had a 50% mandatory overhead in definitioning. Putting my usual operations spin on the problem, I choose to define edge infrastructure in data center management terms. Edge infrastructure has very distinct challenges compared to hyperscale data centers. Read article for the list...

Post 2: Edge Infrastructure Is Not Just Thousands Of Mini Clouds

Running each site as a mini-cloud is clearly not the right answer. There are multiple challenges here. First, any scale infrastructure problem must be solved at the physical layer first. Second, we must have tooling that brings repeatable, automation processes to that layer. It’s not sufficient to have deep control of a single site: we must be able to reliably distribute automation over thousands of sites with limited operational support and bandwidth. These requirements are outside the scope of cloud focused tools.

Post 3: Go CI/CD And Immutable Infrastructure For Edge Computing Management

If “cloudification” is not the solution then where should we look for management patterns? We believe that software development CI/CD and immutable infrastructure patterns are well suited to edge infrastructure use cases. We discussed this at a session at the OpenStack OpenDev Edge summit.

What do YOU think? This is an evolving topic and it’s time to engage in a healthy discussion.

Data Center Bacon: Terraform to Metal with Digital Rebar

Posted on September 19, 2017 by Rob H

TL;DR: We’ve built a buttery smooth Terraform provider for Bare Metal that runs equally on, of course, servers, Packet.net servers or VirtualBox VMs. If you like Hashicorp Terraform and want it to own your data center too, then read on.

Deep into the Digital Rebar Provision (DRP) release plan, a customer asked the RackN team to build a Terraform provider for DRP. They had some very specific requirements that would stress all the new workflows and out-of-band management features in the release: in many ways, this integration is the ultimate proof point for DRP v3.1 because it drives DRP autonomously.

The primary goal was simple: run a data center as a resource pool for Terraform.

Here our CTO, Greg Althaus, giving a short demo of the integration.

Of course, it is not that simple. Operators need to be able to provide plans to pick correct nodes from resources pools. Also, the customer request was to deploy both Linux and Windows images based on Packet. That meant that the system needed both direct-to-disk image writing and cloud-init style post-configuration. The result is deployments that are blazingly fast (sub 5 minutes) and highly portable.

An additional challenge in building the Terraform Provider is that no one wants to practice building plans against actual servers. They are way too slow. We need to be able to build and test the Terraform provider and plans quickly on a laptop or cloud infrastructure like Packet.net. Our solution was to build parallel out-of-band IPMI type plugins for all three platforms so that the Terraform provider could interact with Digital Rebar Provision consistently regardless of the backing infrastructure.

We were able to build a full fidelity CI/CD pipeline for plans without committing dedicated infrastructure at the dev or test phases. That is a significant breakthrough.

Terraform is kicking aaS for cluster deployments on cloud and we’re getting some very enthusiastic responses when we describe both the depth and simplicity of integration with Digital Rebar Provision. We’re actively collecting feedback and testing both new DRP features and Terraform integration so it’s not available for open consumption; however, we very much want to find operators interested in field trials.

Please contact us if Terraform on Metal is interesting. We’d be happy to show you how it works and discuss our next steps.

Further Listening? Our Latest Shiny (L8stSh9y) podcast with Greg Althaus and Stephen Spector covers the work.

Podcast – Terraform and Digital Rebar Provision Bare Metal

Posted on August 31, 2017 by Rob H

In this podcast, Stephen Spector, HPE Cloud Evangelist and Greg Althaus, Co-Founder and CTO RackN, talk about the integration point for Digital Rebar Provisioning with the Terraform solution. The specific focus is on delivering bare metal provisioning to users of Terraform.

About Terraform (LINK)

Terraform enables you to safely and predictably create, change, and improve production infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

More info on Digital Rebar Provisioning

Follow the RackN L8ist Sh9y Podcast

Home Page
RSS Feed
iTunes Feed Coming Soon!

RackN Ends DevOps Gridlock in Data Center [Press Release]

Posted on May 4, 2017 by Rob H

Today we announced the availability of Digital Rebar Provision, the industry’s first cloud-native physical provisioning utility. We’ve had this in the Digital Rebar community for a few weeks before offering support and response has been great!

DR Provision By releasing their API-driven provisioning tool as a stand-alone component of the larger Digital Rebar suite, RackN helps DevOps teams break automation bottlenecks in their legacy data centers without disrupting current operations. The stand-alone open utility can be deployed in under 5 minutes and fits into any data center design. RackN also announced a $1,000 starter support and consulting package to further accelerate transition from tools like Cobbler, MaaS or Stacki to the new Golang utility.

“We were seeing SREs suffering from high job turnover,” said Rob Hirschfeld, RackN founder and CEO. “When their integration plans get gridlocked by legacy tooling they quickly either lose patience or political capital. Digital Rebar Provision replaces the legacy tools without process disruption so that everyone can find shared wins early in large SRE initiatives.”

The first cloud-native physical provisioning utility

Data center provisioning is surprisingly complex because it’s caught between cutting edge hardware and arcane protocols and firmware requirements that are difficult to disrupt. The heart of the system is a fickle combination of specific DHCP options, a firmware bootstrap environment (known as PXE), a very lightweight file transfer protocol (TFTP) and operating system specific templating tools like preseed and kickstart. Getting all these pieces to work together with updated APIs without breaking legacy support has been elusive.

By rethinking physical ops in cloud-native terms, RackN has managed to distill out a powerful provisioning tool for DevOps and SRE minded operators who need robust API/CLI, Day 2 Ops, security and control as primary design requirements. By bootstrapping foundational automation with Digital Rebar Provision, DevOps teams lay a foundation for data center operations that improves collaboration between operators and SRE teams: operators enjoy additional control and reuse and SREs get a doorway into building a fully automated process.

A pragmatic path without burning downing the data center

“I’m excited to see RackN providing a pragmatic path from physical boot to provisioning without having to start over and rebuild my data center to get there.” said Dave McCrory, an early cloud and data gravity innovator. “It’s time for the industry to stop splitting physical and cloud IT processes because snowflaked, manual processes slow everyone down. I can’t imagine an easier on-ramp than Digital Rebar Provision”

The RackN Digital Rebar is making it easy for Cobbler, Stacki, MaaS and Forman users to evaluate our RESTful, Golang, Template-based PXE Provisioning utility. Interested users can evaluate the service in minutes on a laptop or engage with RackN for a more comprehensive trail with expert support. The open Provision service works both independently and as part of Digital Rebar’s full life-cycle hybrid control.

See specific features at http://rackn.com/provision/drsa.

Want help starting on this journey? Contact us and we can help.

DevOps workers, you mother was right: always bring a clean Underlay.

Posted on February 10, 2016 by Rob H

Why did your mom care about underwear? She wanted you to have good hygiene. What is good Ops hygiene? It’s not as simple as keeping up with the laundry, but the idea is similar. It means that we’re not going to get surprised by something in our environment that we’d taken for granted. It means that we have a fundamental level of control to keep clean. Let’s explore this in context.

I’ve struggled with the term “underlay” for infrastructure of a long time. At RackN, we generally prefer the term “ready state” to describe getting systems prepared for install; however, underlay fits very well when we consider it as the foundation for a more building up a platform like Kubernetes, Docker Swarm, Ceph and OpenStack. Even more than single operator applications, these community built platforms require carefully tuned and configured environments. In my experience, getting the underlay right dramatically reduces installation challenges of the platform.

What goes into a clean underlay? All your infrastructure and most of your configuration.

Just buying servers (or cloud instances) does not make a platform. Cloud underlay is nearly as complex, but let’s assume metal here. To turn nodes into a cluster, you need setup their RAID and BIOS. Generally, you’ll also need to configure out-of-band management IPs and security. Those RAID and BIOS settings specific to the function of each node, so you’d better get that right. Then install the operating system. That will need access keys, IP addresses, names, NTP, DNS and proxy configuration just as a start. Before you connect to the wide, make sure to update to your a local mirror and site specific requirements. Installing Docker or a SDN layer? You may have to patch your kernel. It’s already overwhelming and we have not even gotten to the platform specific details!

Buried in this long sequence of configurations are critical details about your network, storage and environment.

Any mistake here and your install goes off the rails. Imagine that your building a house: it’s very expensive to change the plumbing lines once the foundation is poured. Thankfully, software configuration is not concrete but the costs of dealing with bad setup is just as frustrating.

The underlay is the foundation of your install. It needs to be automated and robust.

The challenge compounds once an installation is already in progress because adding the application changes the underlay. When (not if) you make a deploy mistake, you’ll have to either reset the environment or make your deployment idempotent (meaning, able to run the same script multiple times safely). Really, you need to do both.

Why do you need both fast resets and component idempotency? They each help you troubleshoot issues but in different ways. Fast resets ensure that you understand the environment your application requires. Post install tweaks can mask systemic problems that will only be exposed under load. Idempotent action allows you to quickly iterate over individual steps to optimize and isolate components. Together they create resilient automation and good hygiene.

In my experience, the best deployments involved a non-recoverable/destructive performance test followed by a completely fresh install to reset the environment. The Ops equivalent of a full dress rehearsal to flush out issues. I’ve seen similar concepts promoted around the Netflix Chaos Monkey pattern.

If your deployment is too fragile to risk breaking in development and test then you’re signing up for an on-going life of fire fighting. In that case, you’ll definitely need all the “clean underware” you can find.

Full Metal DevOps: 12 things we needed beyond Cobbler

Posted on February 3, 2016 by Rob H

Almost a manifesto!

Rob Hirschfeld

The RackN team did not plan to replace Cobbler, we just needed something that responded to our need for full-cycle cross-platform DevOps automation.

Provisioning an O/S is never enough! You need to coordinate a lot of operational activity to deploy a multi-node system, like OpenStack, Kubernetes, Docker Swarm or Ceph. Since we believe an automated upgrade path is also required, there is a huge gap in provisioning.

So what was needed? Here’s our (rather long!) list of gaps to fill for full Metal DevOps provisioning:

Gap	Commentary
1	Needs to work with Cobbler!	Improve? Yes. Disrupt? Hell No! It has to be OK to leave Cobbler in place while we do something better. I’d be OK to tweak my Cobber to point it to the new stuff.
2	REST API & JSON CLI	Beyond the obvious API, we really want a way to write scripts that drive deployment proactively.
3	Modular Components	If…

View original post 358 more words

Smaller Nodes? Just the Right Size for Docker!

Posted on January 26, 2016 by Rob H

Container workloads have the potential to redefine how we think about scale and hosted infrastructure.

Last Fall, Ubiquity Hosting and RackN announced a 200 node Docker Swarm cluster as a phase one of our collaboration. Unlike cloud-based container workloads demonstrations, we chose to run this cluster directly on the bare metal.

Why bare metal instead of virtualized? We believe that metal offers additional performance, availability and control.

With the cluster automation ready, we’re looking for customers to help us prove those assumptions. While we could simply build on many VMs, our analysis is the a lot of smaller nodes will distribute work more efficiently. Since there is no virtualization overhead, lower RAM systems can still give great performance.

The collaboration with RackN allows us to offer customers a rapid, repeatable cluster capability. Their Digital Rebar automation works on a broad spectrum of infrastructure allow our users to rehearse deployments on cloud, quickly change components and iteratively tune the cluster.

We’re finding that these dedicated metal nodes have much better performance than similar VMs in AWS? Don’t believe us – you can use Digital Rebar to spin up both and compare. Since Digital Rebar is an open source platform, you can explore and expand on it.

The Docker Swarm deployment is just a starting point for us. We want to hear your provisioning ideas and work to turn them into reality.

12 Predictions for ’16: mono-cloud ambitions die as containers drive more hybrid IT

Posted on December 31, 2015 by Rob H

I expect 2016 to be a confusing year for everyone in IT. For 2015, I predicted that new uses for containers are going to upset cloud’s apple cart; however, the replacement paradigm is not clear yet. Consequently, I’m doing a prognostication mix and match: five predictions and seven items on a “container technology watch list.”

TL;DR: In 2016, Hybrid IT arrives on Containers’ wings.

Considering my expectations below, I think it’s time to accept that all IT is heterogeneous and stop trying to box everything into a mono-cloud. Accepting hybrid as current state unblocks many IT decisions that are waiting for things to settle down.

Here’s the memo: “Stop waiting. It’s not going to converge.”

2016 Predictions

Container Adoption Seen As Two Stages: We will finally accept that Containers have strength for both infrastructure (first stage adoption) and application life-cycle (second stage adoption) transformation. Stage one offers value so we will start talking about legacy migration into containers without shaming teams that are not also rewriting apps as immutable microservice unicorns.
OpenStack continues to bump and grow. Adoption is up and open alternatives are disappearing. For dedicated/private IaaS, OpenStack will continue to gain in 2016 for basic VM management. Both competitive and internal pressures continue to threaten the project but I believe they will not emerge in 2016. Here’s my complete OpenStack 2016 post?
Amazon, GCE and Azure make everything else questionable. These services are so deep and rich that I’d question anyone who is not using them. At least one of them simply have to be part of everyone’s IT strategy for financial, talent and technical reasons.
Cloud API becomes irrelevant. Cloud API is so 2011! There are now so many reasonable clients to abstract various Infrastructures that Cloud APIs are less relevant. Capability, interoperability and consistency remain critical factors, but the APIs themselves are not interesting.
Metal aaS gets interesting. I’m a big believer in the power of operating metal via an API and the RackN team delivers it for private infrastructure using Digital Rebar. Now there are several companies (Packet.net, Ubiquity Hosting and others) that offer hosted metal.

2016 Container Tech Watch List

I’m planning posts about all these key container ecosystems for 2016. I think they are all significant contributors to the emerging application life-cycle paradigm.

Service Containers (& VMs): There’s an emerging pattern of infrastructure managed containers that provide critical host services like networking, logging, and monitoring. I believe this pattern will provide significant value and generate it’s own ecosystem.
Networking & Storage Services: Gaps in networking and storage for containers need to get solved in a consistent way. Expect a lot of thrash and innovation here.
Container Orchestration Services: This is the current battleground for container mind share. Kubernetes, Mesos and Docker Swarm get headlines but there are other interesting alternatives.
Containers on Metal: Removing the virtualization layer reduces complexity, overhead and cost. Container workloads are good choices to re-purpose older servers that have too little CPU or RAM to serve as VM hosts. Who can say no to free infrastructure?! While an obvious win to many, we’ll need to make progress on standardized scale and upgrade operations first.
Immutable Infrastructure: Even as this term wins the “most confusing” concept in cloud award, it is an important one for container designers to understand. The unfortunate naming paradox is that immutable infrastructure drives disciplines that allow fast turnover, better security and more dynamic management.
Microservices: The latest generation of service oriented architecture (SOA) benefits from a new class of distribute service registration platforms (etcd and consul) that bring new life into SOA.
Paywall Registries: The important of container registries is easy to overlook because they seem to be version 2.0 of package caches; however, container layering makes these services much more dynamic and central than many realize. (more? Bernard Golden and I already posted about this)

What two items did not make the 2016 cut? 1) Special purpose container-focused operating systems like CoreOS or RancherOS. While interesting, I don’t think these deployment technologies have architectural level influence. 2) Container Security via VMs. I’m seeing patterns where containers may actually be more secure than VMs. This is FUD created by people with a vested interest in virtualization.

Did I miss something? I’d love to know what you think I got right or wrong!

¡Sí, Sí! That’s a Two Hundred Node Metal Docker Swarm Deployment

Posted on November 16, 2015 by Rob H

Today, RackN and Ubiquity Hosting announced a 200 node Docker Swarm deployment on hosted bare metal.

Leveraging the current Digital Rebar core and the RackN Swarm workload, this reference deployment was automatically configured using the same components that also work on a desktop VM deployment. That high fidelity deployment allows operators to start learning quickly on small systems then grow to AWS and if warranted, potentially smoothly transition to scale metal.

This deployment represents RackN starting a new chapter with Digital Rebar because it demonstrates a commitment deploy on any infrastructure: cloud, metal or something in between.

The RackN team started this journey with a “composable ops” vision that allows operators to mix and match. That spans both vendor physical resources and software components such as operating systems, software defined networking and platforms. In the 200 node Swarm cluster, physical infrastructure is provisioned by Ubiquity Hosting not Digital Rebar or RackN. Historically, RackN focused on private infrastructure. Now, users get the option of best-in-class metal deployment without having to own the infrastructure.

We experienced the futility of making Ops homogeneous and declared defeat.

Accepting that each data center has individual Ops was pivotal. Digital Rebar embraces heterogeneity at the most fundamental architectural level. Our system approach and unique composable abstractions allow users to make deployments portable between any infrastructure with existing tooling and operational processes. Portability means that we can both eliminate the fidelity gap as we scale and between deployments.

When multiple scales and sites can share deployment automation, we can finally work together on addressing critical operational issues like scale, high availability and upgrade

This 200 node deployment demonstrates more than scale and the deployment of the latest Docker technology. It is a milestone on the path toward sharable production operations.