Erlang Consul Client

OpenCrowbar has been using Consul more and more deeply.  We’ve reached the point where we must register services on Consul to pass automated tests.

Consequently, I had to write a little Consul client in Erlang.

The client is very basic, but it seems to perform all of the required functions.  It relies on some other libraries in OpenCrowbar’s BDD but they are relatively self-contained.  Pulls welcome if you’d like to help build this out.

Here’s the code and the API reference.   Check in OpenCrowbar/BDD for latest updates and dependencies.

Erlang HTTP client RESTful API Post (example code)

Sometimes I just need to feed the SEO monster for Erlang.  In this case, I could not find a good post that described doing an Erlang http:request that posts with a json payload.  This is to support the BDD (“Cucumber but really fast”) testing framework that I’ve included in Crowbar.

Method = post,
URL = "http://192,168.124.10:3000/node/2.0/new",
Header = [],
Type = "application/json",
Body = "{\"name\":\"\"}",
HTTPOptions = [],
Options = [],
R = httpc:request(Method, {URL, Header, Type, Body}, HTTPOptions, Options),
{ok, {{"HTTP/1.1",ReturnCode, State}, Head, Body}} = R.

{"id", Key} = lists:keyfind("id",1,json:parse(Body)),

Note: See the full source:

The Body will contain your JSON response and I used our Erlang json lib to parse that to find the ID.

In the actual BDD code, it’s a little more complex because I also use our Erlang digest auth.  I hope this helps you!

Opscode Summit Recap – taking Chef & DevOps to a whole new level

Opscode Summit Agenda created by open space

I have to say that last week’s Opscode Community Summit was one of the most productive summits that I have attended. Their use of the open-space meeting format proved to be highly effective for a team of motivated people to self-organize and talk about critical topics. I especially like the agenda negations (see picture for an agenda snapshot) because everyone worked to adjust session times and locations based on what else other sessions being offered. Of course, is also helped to have an unbelievable level of Chef expertise on tap.


Overall, I found the summit to be a very valuable two days; consequently, I feel some need to pay it forward with some a good summary. Part of the goal was for the community to document their sessions on the event wiki (which I have done).

The roadmap sessions were of particular interest to me. In short, Chef is converging the code bases of their three products (hosted, private and open). The primary change on this will moving from CouchBD to a SQL based DB and moving away the API calls away from Merb/Ruby to Erlang. They are also improving search so that we can make more fine-tuned requests that perform better and return less extraneous data.

I had a lot of great conversations. Some of the companies represented included: Monster, Oracle, HP, DTO, Opscode (of course), InfoChimps, Reactor8, and Rackspace. There were many others – overall >100 people attended!

Crowbar & Chef

Greg Althaus and I attended for Dell with a Crowbar specific agenda so my notes reflect the fact that I spent 80% of my time on sessions related to features we need and explaining what we have done with Chef.

Observations related to Crowbar’s use of Chef

  1. There is a class of “orchestration” products that have similar objectives as Crowbar. Ones that I remember are Cluster Chef, Run Deck, Domino
  2. Crowbar uses Chef in a way that is different than users who have a single application to deploy. We use roles and databags to store configuration that other users inject into their recipes. This is dues to the fact that we are trying to create generic recipes that can be applied to many installations.
  3. Our heavy use of roles enables something of a cookbook service pattern. We found that this was confusing to many chef users who rely on the UI and knife. It works for us because all of these interactions are automated by Crowbar.
  4. We picked up some smart security ideas that we’ll incorporate into future versions.

Managed Nodes / External Entities

Our primary focus was creating an “External Entity” or “Managed Node” model. Matt Ray prefers the term “managed node” so I’ll defer to that name for now. This model is needed for Crowbar to manage system components that cannot run an agent such as a network switch, blade chassis, IP power distribution unit (PDU), and a SAN array. The concept for a managed node is that that there is an instance of the chef-client agent that can act as a delegate for the external entity. I had so much to say about that part of the session, I’m posting it as its own topic shortly.

Why I love erlang – a mini recursive JSON parser

As a mental break and to support my erlang version of Cucumber (“BravoDelta”), I spent a little time building out a JSON parser.

Some notes before the code:

  • I could have done it without the case statements (using pattern matching in the functions) but I felt the code was not as readable and there were some cases where I needed the RAW input.
  • I used records because it was important to return BOTH the list and the remaining text.  It also improves the readability if you follow know the syntax (#json = new record, JSON#json = existing)
  • Has minimal error checking – fails = good in a BDD tool
  • Assumed that keys are “safe” words (don’t really need quotes)

Here’s the code.  Enjoy!

Note 2013-11-15: Here’s the active source for this on github.

-record(json, {list=[], raw=[]}).
-record(jsonkv, {value=[], raw=[]}).
% handles values that are quoted (this one ends the quote)
json_value_quoted(Value, [$" | T]) ->
  #jsonkv{value=Value, raw=T};

json_value_quoted(Value, [Next | T]) ->
  json_value_quoted(Value ++ [Next], T).

% returns JSON Key Values with remaining JSON
json_value(Value, RawJSON) ->
  [Next | T] = RawJSON, 
  case Next of
    $: -> throw('unexpected token');
    ${ -> J = json(RawJSON),                                  % recurse to get list
            #jsonkv{value=J#json.list, raw=J#json.raw};  
    $, -> #jsonkv{value=string:strip(Value), raw=RawJSON};    % terminator, return
    $} -> #jsonkv{value=string:strip(Value), raw=RawJSON};    % terminator, return
    $" -> json_value_quoted(Value, T);                        % run to next quote,exit
    _ -> json_value(Value ++ [Next], T)                       % recurse
% parses the Key Value pairs (KVPs) based on , & } delimiters
json(JSON, Key) ->
  [Next | T] = JSON#json.raw,
  case {Next, T} of
    {$", _} -> json(JSON#json{raw=T}, Key);        % ignore
    {${, _} -> json(#json{raw=T}, []);             % start new hash
    {$,, _} -> json(JSON#json{raw=T}, []);         % add new value
    {$:, _} -> KV = json_value([], T),  % get value for key
            List = lists:merge(JSON#json.list, [{string:strip(Key), KV#jsonkv.value}]),
            json(#json{list=List, raw=KV#jsonkv.raw}, []);  % add new KVP
    {$}, []} -> JSON#json.list;                    %DONE!
    {$}, _} -> JSON#json{raw=T};                   %List parse, but more remains!
    {_, _} -> json(JSON#json{raw=T}, Key ++ [Next])  % add to key
% entry point
json(RawJSON) ->
  json(#json{raw=RawJSON}, []).