I’ve started to explore mono, with a view to moving some of our web applications to Linux. Used MonoDevelop  on OSX to  spike a simple HttpHandler to return a response.  I was more interested in how the hosting and deployment story worked with mono.

This is a little list of things I discovered as I went along.

http://www.mono-project.com/ASP.NET has  list of the hosting options available.  Went with the Nginx option.  Mono comes with xsp, which is useful for local testing.

Running a simple web application

To run xsp  /usr/bin/xsp –port 9090 –root  <path to your application>, and the application will be available on http://localhost:9090

To install Nginx on OSX,  get Homebrew.  And then simply  sudo brew install nginix

Follow the instructions here http://www.mono-project.com/FastCGI_Nginx to configure Nginix to work with Mono’s FastCGI server.

On OSX, the Nginix configs can be found in /usr/local/etc/nginx/nginx.conf

This is the configuration I tried for my testing,

In /usr/local/etc/nginx/nginx.conf

server{
   listen 80;
   server_name localhost;
   access_log /var/log/nginx/localhost_mono_access.log;
   location / {
        root /Users/hibri/Projects/WebApp/;
        index default.aspx index.html;
        fastcgi_index default.aspx;
        fastcgi_pass 127.0.0.1:9000;
        include /usr/local/etc/nginx/fastcgi_params;
   }
}

Add the following lines to /usr/local/etc/nginx/fastcgi_params

 fastcgi_param  PATH_INFO          "";
 fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;

Start Nginx.

Start the Mono FastCGI server

fastcgi-mono-server2

     /applications=localhost:/:/Users/hibri/Projects/WebApp/ /socket=tcp:127.0.0.1:9000

And the application is available on http://localhost

Web Frameworks

We use OpenRasta for the services I want to run on Linux. OR didn’t work out of the box. This is something I’ll be exploring in the next few days.

Tried ServiceStack too, and was able to get one our projects (https://github.com/gregsochanik/basic-servicestack-catalogue) working on Mono as is.  Nancy is next on the list.

Then download the F# CTP zip, from here

http://msdn.microsoft.com/en-us/fsharp/cc835251

and extract  it.

Download the mono strong name key to the same place where you’ve extracted F# to.

Run sudo ./install-mono.sh

There is also mono develop f# addin  but is currently broken with mono develop 2.6

Node.js

The session on node.js on Sunday, was my first serious introduction to server side js and node.js.

Start by downloading the source at http://nodejs.org/#download, extract the source do ./configure and make install in the source directory. Takes a few minutes to build. The build works painlessly on OSX and Linux. If you are on OSX install it via brew

http://shapeshed.com/journal/setting-up-nodejs-and-npm-on-mac-osx/

The documentation is at http://nodejs.org/docs/v0.4.8/api/

Node.js is a good way to get into event driven non-blocking programming. It’s easy to do this when you think about doing things (sending responses, rendering) only when things happen.

For example, when data arrives on a socket listening on the server an event is triggered. Code is executed when events are triggered. Instead of polling and waiting for stuff to happen, which can be very in-efficient.

This got me thinking about tiny programs running in a system and only doing things as a result of something being triggered. This could lead to us writing code that is only needed by the system. Code that is not used by the system (i.e not triggered) are culled.

Treating JavaScript as a programming language.

Going on with the js theme, the guys from Caplin Systems, showed how to build applications with js while still testing the full stack. We were shown how to use Eclipse and JSTestDriver. We were also taken thorough building the full application stack using  domain and view models in js. While using a Model View View Model pattern with knockout.js to bind the domain to client HTML.

Master of Puppets

I had mixed feelings about attending this session but in the end it was worth it. Puppet is an open source platform to manage systems, similar to Chef. Puppet and Chef use recipes to build and configure machines. It seems to work smoothly with Ubuntu, using apt get to install and configure the software as specified in a recipe. Still no good Windows support though, which is going to make it hard to use at work.

It is also possible to use Puppet to control/build virtual machines using vagrant. There is also a VMware API and a ruby gem for the API . For further reading on this please follow the links below.

http://www.jedi.be/blog/2009/11/17/controlling-virtual-machines-with-an-API/

http://rubyvmware.rubyforge.org/

Non-Technical Sessions

I enjoyed the non-techy sessions very much. To start it off there was Rachel Davies’ session on building trust within teams. The slides from this are up now http://www.agilexp.com/presentations/SPA-ImprovingTrustInTeams.pdf

Benjamin Mitchell’s session on double loop learning was insightful. It made me think about how much my perception of things don’t necessarily reflect the reality. It is better to seek knowledge than take actions based on assumptions. It became clear how easily we can fall into this trap. There is more reading on double loop learning and the work by Chris Argyris here http://bit.ly/Argyris

Developer anarchy at forward clearly illustrated that to go faster, you need great devs and ditch technology that is slowing down feedback loops. It’s not just about building feedback loops, its how fast you can react to those loops is what matters. On the Internet scale, we’d would have to respond in seconds, minutes and at least in hours. Definitely not in days, sprints or months. In the conversation afterwards, learnt that they spent about 6 months re-building the tools and infrastructure that let them deliver at the speed that the do now.

Comic Communication and Collaboration completed the SPA experience with much hilarity and fun. Think I could try my hand and xkcd style comics. More importantly, the insight learned was ,communicate by talking with peers, or communicate by producing something (i.e readable code, working software). If you have to communicate via email or worse through a 3rd party (i.e project manager) don’t bother. It’s not as effective as you think.

All in all, a very well organised conference, including the invited rants. Looking forward to next year.

Many thanks to those who organised it.

There are tests (mostly what we call acceptance tests). The system under test (SUT) works with a couple of web services to do its work. The problem I’m faced with is that, when I write these tests, the external web services have to be arranged with data for the test, or the test has to rely on existing data. Writing these tests is extremely painful. Knowledge of the magic existing data is required and in the end what we are really writing are integration tests. But we don’t want integration tests.

At 7digital, we are exposing more of our domain via HTTP APIs, in a “NotSOA” manner. To test each service by itself it becomes necessary to mock the dependencies.

Solutions.

There are a couple of solutions to this.
Set up an stub HTTP web service somewhere, and let it return canned responses. It behaves like the real web service, but only returns responses that have been arranged. The disadvantage of this approach is that I have to know about what canned responses have already been setup.

To change the response for a particular test I have to make changes to the stub server and deploy it, as it is a separate application. It takes the focus away from writing the test I’m concerned with.

Another way is to insert some sort of “switch” in production code that will return canned responses when under test. I don’t like this approach because it requires production code just for tests

My solution.

What I want to do is something similar to setting up mocks/stubs in unit tests, but to do it with an actual http server. To setup the stubbed responses in the test code itself, and not to have to make any change to production code, other than a configuration change.

So this is what I came up with

 1: [Test]

 2:         public void SUT_should_return_stubbed_response() {

 3:             IStubHttp stubHttp = HttpMockRepository

 4:                 .At("http://localhost:8080/someapp");

 5: 

 6: 

 7:             const string expected = "<xml><>response>Hello World</response></xml>";

 8:             stubHttp.Stub(x => x.Get("/someendpoint"))

 9:                 .Return(expected)

 10:                 .OK();

 11: 

 12:             string result = new SystemUnderTest().GetData();

 13: 

 14:             Assert.That(result, Is.EqualTo(expected));

 15: 

 16:         }

HttpMockRepoisitory.At creates a HTTP server listening on port 8080, and behaves as if it is process request under the /someapp path. This is the web service that the SUT will get it’s data from.

Using the object returned,  it is possible to setup stubbed responses using a fluent syntax. The stub server can return text and files. I’ve posted a few more examples on github http://github.com/hibri/HttpMock/blob/master/src/HttpMock.Integration.Tests/HttpEndPointTests.cs

Kayak.

I’m using Kayak, a light weight, asynchronous HTTP server written in C#. Kayak, can accept request processing delegates, and post them to the HTTP server listening on the given socket. This allows me to add stub responses at runtime.

Current status.

This is very much a work in progress. HTTP GET works. There is support for returning stubbing content types and HTTP return codes. I’ll be able to add to this while changing a very large test suite to not rely on real web services.  I’ve created a repository on github at http://github.com/hibri/HttpMock

There are no unit tests now, but I’ll be adding them soon as I wanted to prove the concept first.

Describing this as mocking is not entirely correct, but I couldn’t find a term to describe the concept. It is possible to do the same in Ruby using Sinatra.

Found a fairly good set of coursework to get started at http://courses.csail.mit.edu/iphonedev/ . This is a very basic introductory course and the set of presentations guide you through developing a complete iPhone application. Before this I had no clue on how to use XCode. This helped me grasp the basic language concepts. Going through the whole set is highly recommended.

There is a very handy Obj-C tutorial at http://cocoadevcentral.com/d/learn_objectivec/

Setting up tests was frustrating in XCode 3. Although XCode4 has improved on this, it is no where near Eclipse and Visual Studio. Skip the built in test framework (STAssert) in favour of OCHamcrest and you’ll be in familiar territory. There was a bit of hair pulling in figuring out how to get XCode4 to use it.

Now that I’ve figured out XCode4 , I’m going through the  iOS development videos at http://developer.apple.com/videos/iphone/ .

What is a web service wrapper ?

A wrapper for a web service is a library, helps you deal with said service in the language programming language of your choice. It hides the details of the web service, and saves you the trouble of having to know how to parse XML or JSON. The wrapper gives you first class objects to work with.

Many web service providers provide a wrapper for their services in most programming languages.

Why I don’t like wrappers.

I strongly believe that web services should be simple to use. If you expose a web service via HTTP, your consumers should be able to use any HTTP client to consume the service.

You should be able to use a web service by simply typing the URL for the web service method in the web browser’s address bar and see the result in the browser itself.  You should be able to use a command line tool such as curl to call the web service. Using a web service should not be more complicated than this. If you were hardcore even telnet should suffice.

To consume the service in code, the bare minimum a  developer should need is a decent HTTP client library and a standard XML/JSON parser. Even a decent string library should suffice to make sense of the HTTP responses. These are pretty much available in the framework provided out of the box with the major programming languages. Of course there are situations where you’ll need more, but then this should be the exception and not the norm.

From the point of view of a web service provider, this simplicity increases adoption of your web service. Consumers don’t have to wait for you publish a new version of the wrapper library in order to start using a new service endpoint. Maintenance of the wrapper library is a non-issue, as you can focus on fixing issues with the service only, and not the wrapper library.

Avoid using wrappers internally.

When building a web service, avoid the temptation to use wrappers in your acceptance and integration tests. Strongly typed wrappers are a bad idea. I’ve seen this first hand when writing tests when building the 7digital API. Don’t even parse responses to strongly typed objects. I’ve forbidden the use of wrappers and strongly typed objects for testing the 7Digital API within my team.

The reason for this is, as a provider, you have to use the service as a consumer would. Wrappers hide the complexity of your own service, and you won’t know how complex the service has become.  When you work with bare HTTP response strings, you will see potential usability issues that consumers will face.

Publish sample code, but not wrappers.

If you are providing a web service, my recommendation is to publish sample code, and not wrappers around your service. Show developers how to consume the service in their favorite programming languages. A good idea is to give them tests as Martin Fowler recommends. The tests can serve as sample code. They can run those tests against your service and see where the problem lies.

Thoughts

In my experience, using a strongly typed language such as C# has been a bad idea.  Dynamic languages like Ruby can be used to write more tolerant wrappers. This is because with Ruby you can evaluate the API responses at run time rather than having to use an object that requires the response to be in a certain format.

How do we get past the discovery phase without getting stuck ?

My suggestion is to start the tests with what you know and go with it. Think about it as we write the test. It will become clearer gradually.

I’ll explain with an example.

The test name. This is the part where most of us get stuck. It’s an art. Getting it right is solving half the problem, but lets not get stuck on it. It doesn’t matter if we don’t have the perfect test name at the start. Give it a name that is good enough.

[Test]
public void Foo () {

}

This is all we need for now.

We have name, we have a test, our test runner can run the test.

Next. What is the proper outcome to test for ?.  In other words what are we testing ?

Start with the assert first.  Assume we want the title property in some object set to the expected value. Don’t worry about what type the object is, if you don’t have it already.

[Test]
public void Foo (){

  Assert.That(fooObject.Title, Is.EqualTo(expectedTitle));

}

Start with the smallest thing you can assert. Let’s not be concerned with all the other things that need to be tested. In the example code above, I’m testing that an object named fooObject has it’s Title property set to the expected string. At this point I’m not concerned about the exact value of the expectation.

We have an assert. Something needs to happen in the thing we are testing. This is the action. Think about where we get fooObject from.

Some action needs to be executed to get an instance. This action is a method. In code , the way to make something happen is to call a method.

Create a method. Let’s not get stuck on the name too much. I’ll call it Get(). Calling Get() gives me fooObject.

[Test]
public void Foo (){

   var fooObject = controller.Get();
  Assert.That(fooObject.Title, Is.EqualTo(expectedTitle));

}

We have an assert and an action. The code even compiling yet. Keep typing.

Our object controller needs to be an instance of something. Is this an existing object we are testing ?, if not I’ll create it.  We always have a context in which we are writing a unit test. This gives us an indication on what the thing we are testing should be called.

In this example, we are testing a Controller. Instantiate the controller object.

[Test]
public void Foo (){

 var controller = new HomeController();

  var fooObject = controller.Get();  

 Assert.That(fooObject.Title, Is.EqualTo(expectedTitle));

}

Next use resharper ( or similar tool for your language) to create the missing objects. Let’s make the code compile at this time. Give a value for the expectation. Don’t worry about which project (in Visual Studio) the classes will be in. It’s fine to keep it in the test class for now.

[Test]
public void Foo (){

string expectedTitle = "some title";
 var controller = new HomeController();

  var fooObject = controller.Get();  

 Assert.That(fooObject.Title, Is.EqualTo(expectedTitle));

}

We have an assert, action and arrange. Read it the other way.  Arrange, Act and Assert. The 3 parts we need for a test.

Run the test, see it fail. Write the bare minimum of code to make the test pass. Even hardcode things. Get a green bar. Even give temporary names.

Let’s refactor.  Rename things. The test is much more clearer now. Rename Foo to ShouldSetTitleOnView. 

Run the tests again, and keep renaming things. Move files to where they should be.

At this point we can think about injecting dependencies. Remember those hard coded values we put in to make the test pass ?

Inject a dependency to give those values. There is now a place to inject dependencies, the constructor of the HomeController.

To summarize,

Writing the assert first, gives a focal point for the test. Working backwards from this focal point ensures that we write the bare minimum code needed to make the test pass. We don’t need to worry about injecting dependencies that are not even relevant to what we are asserting.

Avoid thinking about your team/company standards of how you should write your tests. Just write the test with what you know. You can always rename things later when you have a green bar to the appropriate standard.

You can follow the same pattern even in unit tests, integration tests or acceptance tests. By doing this we speed up our discovery phase by writing the tests.

I’ve lead 3 teams, all working on core and large products for the companies I’ve worked for. My current and 3rd team is in the midst of a rebirth and in many ways my 4th team.

No team is the same. What worked for one team does not work for the other. Each team has had different approaches to testing, coding standards and work ethics.

What I’d like to reflect upon is what I did, what worked, what didn’t work. I’m very much interested in what other lead devs do for their teams.

What does a lead developer / team lead do ?

I’m not going to define the role here, only to describe what I’ve had to do as a lead dev so far.

Facilitate standups and retrospectives. Watch out for standup smells

Act as a buffer between product ownership/ project management and the team.

Make the call on technical decisions. Have the technical architect hat handy.

Mentor developers. Enforce TDD, refactoring and clean code practices. I say enforce but not encourage. this is because I believe in these so strongly that I’m not willing let my team do anything else.

Build a team that learns not a team that expects me to tell them what to do. Do regular katas and demos of new tools/techniques we can use.  Do code reviews and refactoring exercises as a team, in front of a projector.

Keep the team focused. I’ve done this by having the team not work on anything else other than what our card wall says.

Know the product/ project thoroughly. Learn the quirks, know where everything is. Know the source, read the source to figure out what is going on.

Own the release process, do releases.

Do the grunt work. As a lead dev, I’ve spent an great deal of time on build and deployment scripts. Setting up source control and continuous integration with Teamcity

Staying out of the way. Leave  the majority of the work to the team. I expect to be pulled into meetings, asked to answer questions and deal with issues outside of the team. Part of this is good, as this takes away distractions from the rest of the team, but this also means that I can’t work on something continuously and see it to the end most of the time.  Let someone else in the team own the work.

Be there for the team, be prepared to answer questions all the time. This could be design decisions, having to explain the domain, or the history of why something was done in a certain way. This can be exhausting but it is necessary. 

Listen to the team chatter. Ask what is going on, when there is a discussion around code or a problem. Listen, observe and break into the conversation if needed.

Know the strengths and weaknesses of each team member. Know what they can do and what they cant do. Manage pair programming, some pairs may not be productive. This is necessary in the early stages of a team, but there will always be differences in skill levels.

What works.

I’ve covered a few of these above, but these the short list below is what I think every team needs.

Fostering a learning culture. The best teams I’ve had learned together and taught each other.

Strong discipline in the team.  Stick to what we’ve agreed as a team.

Having a programming god on the team. An expert, for example who knows NHibernate, MVC or can write a book on TDD, help with technical decisions, and tell you what you are doing is wrong. Someone who can go away and figure out the hard things and come back and present to the team.

Automate automate automate.

Have people with passion for what they do. Encourage this.

What doesn’t work.

Team members with an aversion to pair programming and sharing. No rock stars or heroes. Notice this early, don’t ignore it. This will fester and kill team morale.

Fear of code, and fear of breaking things. People shouldn’t be afraid to make mistakes. Mistakes are ok, we learn by breaking things. Give team members the power and confidence to change things.

Indiscipline. Nip it in the bud. Tell people when they are not following the rules and agreed practices. This is something I’ve learned the hard way.

Self organization without direction. Self organization needs a goal. A team suffers when they don’t have know what they are working towards.

Team democracy is not always good, a benevolent dictatorship works much better.

It’s the team, not the project.

An important lesson I’ve learnt is that, what matters is not the software we produce, it is what the team learns while writing all that code. The code we write is an expression of what we learn, and every new line of code is a new learning opportunity.

A good project, with quality code and a few bugs in production is a side effect of a good team, and this team is the most important asset a company has, and not the software that was produced.

Let’s take a step back. Why do we need a branching strategy ? What is a branch ?

We want to put some code in a source control branch, because the code contains the risk of breaking the software in the mainline of development, trunk.

Are we 100% sure that the code in the branch won’t break what is in trunk ? We won’t know for sure till we integrate the branch with trunk.  We won’t know, until it goes through automated/manual testing, and all this after going through merge hell.

We put code in a branch to reduce risk. However, we haven’t reduced that risk. The risk is still there, and we bring it back into trunk. Why not look at methods of reducing the risk in the first place ?

How do we reduce risk ?

1. Cut up a big piece of risk into smaller pieces of risk.

The bigger the risk the more chopping it needs. Now we have smaller things to work on. We do those small pieces one by one, and if something breaks, we know which change broke it. Easier to fix because it was a small change.

2. Break down a big piece of risk, into parts that have no risk and parts that have risk.

Analyze the problem, break it out in to parts that can be done without breaking our software. This is interesting. The part which we thought would be a problem might have become a non-issue. We’ve isolated it. We know exactly which change will break our software. See 1 above

3. Write tests for things that can be broken by the risky bit of code and continuously test.

We know what could break, by the new change. Before we make the change, let’s write tests for our software so we know when it is broken.  Let’s make a change. Is anything broken ? No. Did the next change break it ? yes. Fix it. Keep going.

Instead of shoving off our risky code into a branch, we’ve learnt to manage the risk, and reduce it. If the risk is so high that we can’t mitigate it by doing 1,2 and 3, then let’s create a branch for the code. We have a branching strategy based on risk. We create a branch only for code that is riskiest, and we haven’t been able to reduce that risk by a divide and conquer approach.

In my opinion this is a much better way than having a default branching strategy for every piece of work/story/feature/MMF. Working continuously on trunk has benefits. We can release a feature faster, get refactoring changes others have made quicker. Not go through merge hell, and risk loosing code in the process. Along the way we’ve learnt how to break up a problem into smaller pieces. We’ve got better at writing tests. We’ve learnt how to structure our software so that one thing does not break everything else and we get closer to the nirvana of continuous deployment, because we have only one production line of code to deploy from.

Do you need a branching strategy ? Think again.

These need a signed XML message, in the POST body. A typical message looks like this

<request>
    <nonce>ABCD123==</nonce>
    <hmac>XXXXXXXXX===</hmac>
    <distributor>uid:8888-43434-34343434</distributor>
   <resource>
     ......
   </resouce>
</request>

The hmac element contains the SHA1 signature  of the XML message. The signature is generated using a shared secret.  The signature is for the whole XML except the hmac element. Before signing the message, we have to construct the XML without the hmac, then sign it, and then add then hmac.

I did this by using a two stage Xml serialization process. This may not be the best way to do it, and there has to be a better solution.  I created a class named SignedXMLSerializer, and this inherits from XmlSerializer. SignedXMLSerializer serializes objects that are of the base type Signable. The Signable class has two properties, Nonce and Hmac. Any class that has to be sent as a signed XML message must inherit from Signable

public abstract class Signable
    {
        [XmlElement("nonce")]
        public string Nonce { get; set; }

        [XmlElement("hmac")]
        public string HMAC { get; set; }
    }

In the serialize method of the SignedXMLSerializer, we first generate the nonce. The nonce makes each message unique. Then we serialize the object to a string. The signature is generated using this string, and assigned it to the Hmac property.

We then serialize the signed object to the writer that was passed in.

public void Serialize(T o, XmlTextWriter writer) {
            o.Nonce = GenerateNonce();
            StringBuilder stringBuilder = GetXMLToSign(o);
            string hmac = GetSignature(stringBuilder);
            o.HMAC = hmac;
            Serialize(writer, o);
        }

To use the SignedXMLSerializer, pass in one of the HashAlgorithm types. I’ve used SHA1 in my case.

This makes it easier to use with different signing algorithms. 

Typical usage of the SignedXMLSerializer is as follows

HMACSHA1 hmacsha1 = new HMACSHA1 {Key = Encoding.ASCII.GetBytes("consumerSecret")};
            SignedXmlSerializer<Request> signedXmlSerializer = new SignedXmlSerializer<Request>(hmacsha1);

 StringBuilder sb = new StringBuilder();
            StringWriter writer = new StringWriter(sb);
            signedXmlSerializer.Serialize(req, new XmlTextWriter(writer));

The final serialized string can be sent via HTTP post to ACS4. 

Code for the SignedXMLSerializer class is here http://gist.github.com/303962