Erlang HTTP client RESTful API Post (example code)

Sometimes I just need to feed the SEO monster for Erlang.  In this case, I could not find a good post that described doing an Erlang http:request that posts with a json payload.  This is to support the BDD (“Cucumber but really fast”) testing framework that I’ve included in Crowbar.

Method = post,
URL = "http://192,168.124.10:3000/node/2.0/new",
Header = [],
Type = "application/json",
Body = "{\"name\":\"foo.example.com\"}",
HTTPOptions = [],
Options = [],
R = httpc:request(Method, {URL, Header, Type, Body}, HTTPOptions, Options),
{ok, {{"HTTP/1.1",ReturnCode, State}, Head, Body}} = R.

{"id", Key} = lists:keyfind("id",1,json:parse(Body)),

Note: See the full source: https://github.com/crowbar/barclamp-crowbar/blob/master/BDD/digest_auth.erl

The Body will contain your JSON response and I used our Erlang json lib to parse that to find the ID.

In the actual BDD code, it’s a little more complex because I also use our Erlang digest auth.  I hope this helps you!

WhatTheBus Map seeds planted, looks like Cucumbers

Today’s WhatTheBus progress (see tag “MapSeeds”) created the foundation for live maps. These maps will only show a single bus at a time because our initial use case is that a parent wants to monitor their child’s bus.

To accomplish this use case, we need to provide a list of all the buses in the system (grouped by district) and then allow users to select a bus and see the map.

This foundation started with a series of Cucumber tests that verified the navigation structure and the base map page.  These pages used the existing page tests (yawn).

In fact, most of this work is pretty dull Rails web navigation stuff.  The interesting parts were:

  1. DRY the bus location from cache by moving it into the model
  2. Add jRails so that we can use jQuery goodness
  3. Optimization: added index for Bus xref
  4. Optimization: added ?cache flag for the bus json requests to bypass database calls on location only requests

In our next pass, we’ll add support for Google Maps (v3).  Until then, we just have some simple script that pulls the current bus position using our existing bus JSON request.  I am using the simulator (“rake sim:move”) to verify this; unfortunately, Cucumber does have native AJAX support.

Here’s the <script>

<script type="text/javascript"> var xref = '<%= @bus.xref %>'; var name = '<%= @bus.name %>'; function initMap() { window.setInterval('updateMap();', <%= MAP_UPDATE_SECS %>000); updateMap(); } function updateMap() { jQuery("#ll").text("?,?"); // get the data for the map jQuery.getJSON("/bus/index/"+ xref +".json?cache", {}, function(data){ jQuery("#ll").text(data.buses[xref].lat + ',' + data.buses[xref].lng); }); } </script>

WhatTheBus fun with Cucumber and MemCacheD

Sometimes a problem has to kick you upside the head so you can learn an important lesson.  Tonight’s head slapper was an interaction between Cucumber and MemCacheD.

If you are using CUCUMBER AND MEMCACHE read this post carefully so you don’t get burned.  If you’re using MemCache and not writing tests then return to Jail, do not collect $200.

It’s important to note that Cucumber has the handy side effect of running each scenario in a transaction.  The impact is that the data from each scenario does not impact the next scenario.  (note: you can pre-load data into cucumber using fixtures).

However, Cucumber does not do any rollback for Cache keys added into MemCache.  In fact, your MemCache entries will happily persist between your development and test systems.

WhatTheBus has a simple check to reduce database writes – it only writes to the database if there is no cache hit for the bus.  My thinking is that we only need to add a new bus if there is no key as shown in this partial snippet:

  cache = Rails.cache.read params[:id]
  if cache.nil?
     bus = Bus.find_or_create_by_xref :name => params[:name], :xref => params[:id]
  end

This works great for live testing, but fails in technicolor for Cucumber because tests with the same ID will not make it to the find_or_create.

To solve the problem, I had to add a pre-condition (‘given’ in Cucumber speak) to each scenario to make sure the cache was cleared.  It looks like this in the scenario feature:

  Given no cache for "1234"

And that’s translated as code in the steps like so:

  Given /^no cache for "([^\"]*)"$/ do |id|
   Rails.cache.delete id
  end

WhatTheBus, Day1: MemCacheD roundtrip

Today I got the very basic bus data collection working using Cucumber TDD.  That means that I wrote the basic test I wanted to prove BEFORE I wrote the code that operates the test.

The Cucumber feature test looks like this:

Feature: Mobile Access
In order to ensure that location updates are captured
School Bus Location providers
want to have data they send stored on the site

Scenario: Update Location
When bus named “lion” in the “eanes” district with a id of “1234″ goes to “32,-97″
When I go to the bus “1234″ page
Then json has an object called “buses”
And json has a record “1234″ in “buses” with “lat” value “32″
And json has a record “1234″ in “buses” with “lng” value “-97″

There’s is some code behind this feature that calls the web page and gets the JSON response back.  The code that actually does the work in the bus controller is even simpler:

The at routine takes location updates just parses the parameters and stuffs it into our cache.  For now, we’ll ignore names and district data.

def at

Rails.cache.write params[:id], “#{params[:lat]},#{params[:lng]},#{params[:name]},#{params[:district]}”, :raw=>:true, :unless_exist => false, :expires_in => 5.minutes
render :nothing => true

end

The code that returns the location (index) pulls the string out of the cache and returns the value as simple JSON.

def index

data = Rails.cache.read(params[:id], :raw => true).split(‘,’)
if data.nil?
render :nothing => true
else
render :json => {:buses => { params[:id].to_sym => { :lat => data[0], :lng => data[1] } } }
end

end

Not much to it!  It’s handy that Rails has memcache support baked right in!  I just had to add a line to the environment.rb file and start my memcached server.

Cloud Reference App, “What The Bus” intro

Today I started working on an application to demonstrate “Cloud Scale” concepts.  I had planned to do this using the PetShop application; unfortunately, the 1995 era PetShop Rails migration would take more repair work then a complete rewrite (HTML tables, no CSS, bad forms, no migrations, poor session architecture).

If I’m considering a fresh start, I’d rather do it with one of my non-PetShop pet projects called “WhatTheBus.”  The concept combines inbound live data feeds and geo mapping with a hyper-scale use target.  The use case is to allow parents to see when their kids’ bus is running late using the phone from the bus stop.

I’m putting the code in git://github.com/ravolt/WhatTheBus.git and tracking my updates on this bog.

My first sprint is to build the shell for this application.  That includes:

  • the shell RAILS application
  • Cucumber for testing
  • MemCacheD
  • Simple test that sets the location of a bus (using a GET, sorry) in the cache and checks that it can retrieve that update.

This sprint does not include a map or any database.  I’ll post more as we build out this app.

Note: http://WhatTheBus.com is a working name for this project because it appeals to m warped sense of humor.  It will likely appear under the sanitary ShowBus moniker: http://showb.us.