Today, I’m presenting this topic (PKI automation & rotation) at Defragcon so I wanted to share this background more broadly as a companion for that presentation. I know this is a long post – hang with me, PKI is complex.
Building automation that creates a secure infrastructure is as critical as it is hard to accomplish. For all the we talk about repeatable automation, actually doing it securely is a challenge. Why? Because we cannot simply encode passwords, security tokens or trust into our scripts. Even more critically, secure configuration is antithetical to general immutable automation: it requires that each unit is different and unique.
Over the summer, the RackN team expanded open source Digital Rebar to include roles that build a service-by-service internal public key infrastructure (PKI).
This is a significant advance in provisioning infrastructure because it allows bootstrapping transport layer security (TLS) encryption without having to assume trust at the edges. This is not general PKI: the goal is for internal trust zones that have no external trust anchors.
Before I explain the details, it’s important to understand that RackN did not build a new encryption model! We leveraged the ones that exist and automated them. The challenge has been automating PKI local certificate authorities (CA) and tightly scoped certificates with standard configuration tools. Digital Rebar solves this by merging service management, node configuration and orchestration.
I’ll try and break this down into the key elements of encryption, keys and trust.
The goal is simple: we want to be able to create secure communications (that’s TLS) between networked services. To do that, they need to be able to agree on encryption keys for dialog (that’s PKI). These keys are managed in public and private pairs: one side uses the public key to encrypt a message that can only be decoded with the receiver’s private key.
To stand up a secure REST API service, we need to create a private key held by the server and a public key that is given to each client that wants secure communication with the server.
Now the parties can create secure communications (TLS) between networked services. To do that, they need to be able to agree on encryption keys for dialog. These keys are managed in public and private pairs: one side uses the public key to encrypt a message that can only be decoded with the receiver’s private key.
Unfortunately, point-to-point key exchange is not enough to establish secure communications. It too easy to impersonate a service or intercept traffic.
Part of the solution is to include holder identity information into the key itself such as the name or IP address of the server. The more specific the information, the harder it is to break the trust. Unfortunately, many automation patterns simply use wildcard (or unspecific) identity because it is very difficult for them to predict the IP address or name of a server. To address that problem, we only generate certificates once the system details are known. Even better, it’s then possible to regenerate certificates (known as key rotation) after initial deployment.
While identity improves things, it’s still not sufficient. We need to have a trusted third party who can validate that the keys are legitimate to make the system truly robust. In this case, the certificate authority (CA) that issues the keys signs them so that both parties are able to trust each other. There’s no practical way to intercept communications between the trusted end points without signed keys from the central CA. The system requires that we can build and maintain these three way relationships. For public websites, we can rely on root certificates; however, that’s not practical or desirable for dynamic internal encryption needs.
So what did we do with Digital Rebar? We’ve embedded a certificate authority (CA) service into the core orchestration engine (called “trust me”).
The Digital Rebar CA can be told to generate a root certificate on a per service basis. When we add a server for that service, the CA issues a unique signed certificate matching the server identity. When we add a client for that service, the CA issues a unique signed public key for the client matching the client’s identity. The server will reject communication from unknown public keys. In this way, each service is able to ensure that it is only communicating with trusted end points.
Wow, that’s a lot of information! Getting security right is complex and often neglected. Our focus is provisioning automation, so these efforts do not cover all PKI lifecycle issues or challenges. We’ve got a long list of integrations, tools and next steps that we’d like to accomplish.
Our goal was to automate building secure communication as a default. We think these enhancements to Digital Rebar are a step in that direction. Please let us know if you think this approach is helpful.
I had the good fortune of lunching with Lachie just before the interview aired. We got compare notes about changes going on in the container space. Some of those insights will end up in my 
In fact, I’d 
Despite his fortune and fame, there was a period in the middle of Prince’s career in which he felt creatively and financially locked-in by the big record companies. Once Prince (and the unpronounceable symbol) broke away from Warner Music, he was able to produce music under his own label. This action enabled him to create music without a major record label dictating when he needed to produce a new album and what it needed to sound like. In addition, he was now able to market his new recordings to the distribution platform that supported his artistic and financial goals. While still having ties to Warner Music, he was no longer bound by their business practices. Along with starting his own music subscription service, Prince cut deals with Arista, Columbia, iTunes and Sony. Prince’s music production had operational portability, business agility and choice (seven Grammy awards and 100 million record sales also help create that kind of leverage.).
We think Kubernetes is an awesome way to run applications at scale! Unfortunately, there’s a bootstrapping problem: we need good ways to build secure & reliable scale environments around Kubernetes. While some parts of the platform administration leverage the platform (cool!), there are fundamental operational topics that need to be addressed and questions (like upgrade and conformance) that need to be answered.
Rob Hirschfeld 