Tag Archives: Continuous Integration

Want CI Consul Love? OK! Run Consul in Travis-CI [example scripts]

Posted on by
1

If you are designing an application that uses microservice registration AND continuous integration then this post is for you!  If not, get with the program, you are a fossil.

Inside The EngineSunday night, I posted about the Erlang Consul client I wrote for our Behavior Driven Development (BDD) testing infrastructure.  That exposed a need to run a Consul service in the OpenCrowbar Travis-CI build automation that validates all of our pull requests.  Basically, Travis spins up the full OpenCrowbar API and workers (we call it the annealer) which in turn registers services in Consul.

NOTE: This is pseudo instructions.  In the actual code (here too), I created a script to install consul but this is more illustrative of the changes you need to make in your .travis.yml file.

In the first snippet, we download and unzip consul.  It’s in GO so that’s about all we need for an install.  I added a version check for logging validation.

before_script:
  - wget 'https://dl.bintray.com/mitchellh/consul/0.4.1_linux_amd64.zip'
  - unzip "0.4.1_linux_amd64.zip"
  - ./consul --version

In the second step, we setup the consul service and register it to itself in the background.  That allows the other services to access it.

script: 
  - ../consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul &

After that, the BDD infrastructure can register the fake services that we expect (I created an erlang consul:reg_serv(“name”) routine that makes this super easy).  Once the services are registered, OpenCrowbar will check for the services and continue without trying to instantiate them (which it cannot do in Travis).

Here’s the pull request with the changes.

Cloud Dev Laptop

Posted on by
Reply
This post is in response to multiple requests I’ve gotten from people outside of Dell.  My apologies if it is too commercial.  I work for Dell  and we make hot laptops AND clouds.
When you’re building clouds (as opposed to cloud applications), you need heavy equipment.  So it’s no surprise that I use a Precision M6600 17″ laptop that is capable of running a complete multi-node cloud data center.
IMHO, here are the core requirements for a Cloud Builder laptop:
  1. SSDs (I have two 1/4 TB of SSD):  We are constantly building/installing operating systems.  These are high I/O activities so SSDs are essential.  I’m constantly on the edge of no free space even with 1/2 TB .
  2. RAM (I have 32 Gb): It’s normal for us to run multiple VMs.  If you RAM starve your VMs (I used to have 16) then they page fault and you’re back to constrained disk I/O.  We assign 4 GB RAM per VM because it’s just faster.
  3. Many Cores: VMs w/ 1 CPU = thread contention.  Adding RAM and Disk can’t fix a threading issues.
  4. Bonus: I like a good keyboard and big display – I code, type & read a lot so the 17″ display helps.
For our devs, a normal cycle is write (desktop) -> build (in a VM) -> deploy (on additional VMs) -> full test requires >4 VMs (that’s over 16 GB RAM).  I don’t want to check in code until I complete that cycle.  On small RAM and spinning HDD  that cycle takes >1 hour.  On my laptop it is <15 minutes!
There are only a few models of laptop that can pack that type of power and they demand a premium; however, the extra umph translates into at least 3 or 4 more full cycles per day.  That’s a whole lot of extra productivity.

OpenStack: Five Challenges & Conference Observations

Posted on by
Reply

I was part of the Dell contingent at the OpenStack design conference earlier in the month.  The conference opened a new chapter for the project because the number of contributing companies reached critical mass.  That means that the core committers are no longer employed by just one or two entities; instead, there are more moneyed interests rubbing elbows and figuring out how to collaborate.

From my perspective (from interview with @Cote ), this changed the tone of the conference from more future looking to pragmatic.

That does not mean that everything is sunshine and rainbows for OpenStack clouds, there are real issues to be resolved.  IMHO, the top issues for OpenStack are:

  1. API implementation vs specification
  2. Building up coverage on continuous integration
  3. Ensuring that we can deploy consistently in multi-node systems
  4. Getting contributions from new members
  5. Figuring out which projects are core, satellite, missing or junk.  [xref 2014 DefCore]

Of these issues, I’ve been reconsidering my position favoring API via Implementation over specification (past position).  This has been a subject of debate on my team at Dell and I like Greg Althaus’ succinct articulation of the problem with implementation driven API: “it is not fair.”  This also ends up being a branding issues for OpenStack because governance needs to figure out which is a “real” OpenStack cloud deployment that can use the brand.  Does it have to be 100% of the source?  What about extensions?  What if it uses the API with an alternate implementation?

Of the other issues, most are related to maturity.  I think #2 needs pressure by and commitment from the larger players (Dell very much included).  Crowbar and the deployment blueprint is our answer #3.  Shouting the “don’t fork it up” chorus from the roof tops addresses contributions while #5 will require some strong governance and inevitably create some hurt feelings.