Want CI Consul Love? OK! Run Consul in Travis-CI [example scripts]

If you are designing an application that uses microservice registration AND continuous integration then this post is for you!  If not, get with the program, you are a fossil.

Inside The EngineSunday night, I posted about the Erlang Consul client I wrote for our Behavior Driven Development (BDD) testing infrastructure.  That exposed a need to run a Consul service in the OpenCrowbar Travis-CI build automation that validates all of our pull requests.  Basically, Travis spins up the full OpenCrowbar API and workers (we call it the annealer) which in turn registers services in Consul.

NOTE: This is pseudo instructions.  In the actual code (here too), I created a script to install consul but this is more illustrative of the changes you need to make in your .travis.yml file.

In the first snippet, we download and unzip consul.  It’s in GO so that’s about all we need for an install.  I added a version check for logging validation.

  - wget 'https://dl.bintray.com/mitchellh/consul/0.4.1_linux_amd64.zip'
  - unzip "0.4.1_linux_amd64.zip"
  - ./consul --version

In the second step, we setup the consul service and register it to itself in the background.  That allows the other services to access it.

  - ../consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul &

After that, the BDD infrastructure can register the fake services that we expect (I created an erlang consul:reg_serv(“name”) routine that makes this super easy).  Once the services are registered, OpenCrowbar will check for the services and continue without trying to instantiate them (which it cannot do in Travis).

Here’s the pull request with the changes.

Erlang Consul Client

OpenCrowbar has been using Consul more and more deeply.  We’ve reached the point where we must register services on Consul to pass automated tests.

Consequently, I had to write a little Consul client in Erlang.

The client is very basic, but it seems to perform all of the required functions.  It relies on some other libraries in OpenCrowbar’s BDD but they are relatively self-contained.  Pulls welcome if you’d like to help build this out.

Here’s the code and the API reference.   Check in OpenCrowbar/BDD for latest updates and dependencies.

OpenStack’s Test Driven Core > it’s where I think “what is core” discussions are heading


core lighthouse

In helping drive OpenStack’s “what is core” dialog, I’ve had the privilege of listening to a lot of viewpoints about what we are and should be.  Throughout the process, I’ve tried to put aside my positions and be an objective listener.  In this post, I’m expressing where I think this effort will lead us.

If OpenStack culture values implementation over API then our core definition should too.

How do we make a core definition that values implementation over API?  I think that our definition should also be based on what’s working in the field over qualitative definitional statements.  The challenge in defining core is to find a way to reinforce this culture in a quantifiable way.

The path forward lies in concrete decomposition (and not because you were talked to death on the sidewalk).

Concrete decomposition means breaking Core into small units for discussion like “is provisioning a single server critical?”  More importantly, we can use tests as the unit for decomposition.  Tests are gold when it comes to defining expected OpenStack behaviors.  In our tests, we have a description of which use-cases have been implemented.  Discussing those use-cases is much more finite than arguing over stable versus innovative development methodologies.

I believe that we are moving toward community tests playing an essential role in OpenStack.  As a believer in the value of BDD and CI, I think that placing high value on tests improves the project in fundamental ways beyond defining Core.  It creates a commercial motivation for contributors to add tests, inches us toward interoperability, and helps drive stability for users.  In these ways, using tests to measure OpenStack drives the right behaviors for the project.

Another consequence I anticipate is a new role for the User Committee (UC).  With a growing body of tests, the OpenStack Foundation needs a way to figure out the subset of tests which are required.  While the Technical Committee (TC) should demand a comprehensive suite of tests for all projects, they lack the perspective to figure out which use-cases are being implemented by our user base.  Gathering that data is already the domain of the UC so asking them to match implemented use-cases to tests seems like a natural extension of their role.

By having data supporting the elevation of tests to must-pass status, I envision a definition of Core that is based on how OpenStack is implemented.  That, in turn, will help drive our broader interoperability objectives.

Erlang HTTP client RESTful API Post (example code)

Sometimes I just need to feed the SEO monster for Erlang.  In this case, I could not find a good post that described doing an Erlang http:request that posts with a json payload.  This is to support the BDD (“Cucumber but really fast”) testing framework that I’ve included in Crowbar.

Method = post,
URL = "http://192,168.124.10:3000/node/2.0/new",
Header = [],
Type = "application/json",
Body = "{\"name\":\"foo.example.com\"}",
HTTPOptions = [],
Options = [],
R = httpc:request(Method, {URL, Header, Type, Body}, HTTPOptions, Options),
{ok, {{"HTTP/1.1",ReturnCode, State}, Head, Body}} = R.

{"id", Key} = lists:keyfind("id",1,json:parse(Body)),

Note: See the full source: https://github.com/crowbar/barclamp-crowbar/blob/master/BDD/digest_auth.erl

The Body will contain your JSON response and I used our Erlang json lib to parse that to find the ID.

In the actual BDD code, it’s a little more complex because I also use our Erlang digest auth.  I hope this helps you!